-- Logs begin at Tue 2022-08-30 01:11:21 UTC, end at Tue 2022-08-30 02:23:38 UTC. -- Aug 30 01:41:41.394936 dv-cinder-819667-6-iscsi systemd[1]: Starting Devstack devstack@keystone.service... Aug 30 01:41:41.404581 dv-cinder-819667-6-iscsi devstack@keystone.service[81563]: [uWSGI] getting INI configuration from /etc/keystone/keystone-uwsgi-public.ini Aug 30 01:41:41.406620 dv-cinder-819667-6-iscsi devstack@keystone.service[81563]: *** Starting uWSGI 2.0.18-debian (64bit) on [Tue Aug 30 01:41:41 2022] *** Aug 30 01:41:41.406620 dv-cinder-819667-6-iscsi devstack@keystone.service[81563]: compiled with version: 10.0.1 20200405 (experimental) [master revision 0be9efad938:fcb98e4978a:705510a708d3642c9c962beb663c476167e4e8a4] on 11 April 2020 11:15:55 Aug 30 01:41:41.406620 dv-cinder-819667-6-iscsi devstack@keystone.service[81563]: os: Linux-5.4.0-72-generic #80-Ubuntu SMP Mon Apr 12 17:35:00 UTC 2021 Aug 30 01:41:41.406620 dv-cinder-819667-6-iscsi devstack@keystone.service[81563]: nodename: dv-cinder-819667-6-iscsi Aug 30 01:41:41.406620 dv-cinder-819667-6-iscsi devstack@keystone.service[81563]: machine: x86_64 Aug 30 01:41:41.406620 dv-cinder-819667-6-iscsi devstack@keystone.service[81563]: clock source: unix Aug 30 01:41:41.406620 dv-cinder-819667-6-iscsi devstack@keystone.service[81563]: pcre jit disabled Aug 30 01:41:41.406620 dv-cinder-819667-6-iscsi devstack@keystone.service[81563]: detected number of CPU cores: 4 Aug 30 01:41:41.406620 dv-cinder-819667-6-iscsi devstack@keystone.service[81563]: current working directory: / Aug 30 01:41:41.406620 dv-cinder-819667-6-iscsi devstack@keystone.service[81563]: detected binary path: /usr/bin/uwsgi-core Aug 30 01:41:41.406620 dv-cinder-819667-6-iscsi devstack@keystone.service[81563]: your processes number limit is 47680 Aug 30 01:41:41.407326 dv-cinder-819667-6-iscsi devstack@keystone.service[81563]: your memory page size is 4096 bytes Aug 30 01:41:41.407326 dv-cinder-819667-6-iscsi devstack@keystone.service[81563]: detected max file descriptor number: 2048 Aug 30 01:41:41.407326 dv-cinder-819667-6-iscsi devstack@keystone.service[81563]: lock engine: pthread robust mutexes Aug 30 01:41:41.407326 dv-cinder-819667-6-iscsi devstack@keystone.service[81563]: thunder lock: enabled Aug 30 01:41:41.407326 dv-cinder-819667-6-iscsi devstack@keystone.service[81563]: uwsgi socket 0 bound to UNIX address /var/run/uwsgi/keystone-wsgi-public.socket fd 4 Aug 30 01:41:41.407326 dv-cinder-819667-6-iscsi devstack@keystone.service[81563]: Python version: 3.8.10 (default, Jun 22 2022, 20:18:18) [GCC 9.4.0] Aug 30 01:41:41.447436 dv-cinder-819667-6-iscsi devstack@keystone.service[81563]: Python main interpreter initialized at 0x562424ee09c0 Aug 30 01:41:41.447436 dv-cinder-819667-6-iscsi devstack@keystone.service[81563]: python threads support enabled Aug 30 01:41:41.447436 dv-cinder-819667-6-iscsi devstack@keystone.service[81563]: your server socket listen backlog is limited to 100 connections Aug 30 01:41:41.447436 dv-cinder-819667-6-iscsi devstack@keystone.service[81563]: your mercy for graceful operations on workers is 90 seconds Aug 30 01:41:41.447857 dv-cinder-819667-6-iscsi devstack@keystone.service[81563]: mapped 403077 bytes (393 KB) for 2 cores Aug 30 01:41:41.447944 dv-cinder-819667-6-iscsi devstack@keystone.service[81563]: *** Operational MODE: preforking *** Aug 30 01:41:41.448768 dv-cinder-819667-6-iscsi devstack@keystone.service[81563]: *** uWSGI is running in multiple interpreter mode *** Aug 30 01:41:41.448768 dv-cinder-819667-6-iscsi devstack@keystone.service[81563]: spawned uWSGI master process (pid: 81563) Aug 30 01:41:41.448980 dv-cinder-819667-6-iscsi systemd[1]: Started Devstack devstack@keystone.service. Aug 30 01:41:41.450160 dv-cinder-819667-6-iscsi devstack@keystone.service[81563]: spawned uWSGI worker 1 (pid: 81577, cores: 1) Aug 30 01:41:41.450160 dv-cinder-819667-6-iscsi devstack@keystone.service[81563]: spawned uWSGI worker 2 (pid: 81578, cores: 1) Aug 30 01:41:41.450160 dv-cinder-819667-6-iscsi devstack@keystone.service[81563]: running "unix_signal:15 gracefully_kill_them_all" (master-start)... Aug 30 01:41:45.820382 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: /usr/lib/python3/dist-packages/secretstorage/dhcrypto.py:15: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead Aug 30 01:41:45.820382 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: from cryptography.utils import int_from_bytes Aug 30 01:41:45.821163 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: /usr/lib/python3/dist-packages/secretstorage/dhcrypto.py:15: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead Aug 30 01:41:45.821163 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: from cryptography.utils import int_from_bytes Aug 30 01:41:45.821626 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: /usr/lib/python3/dist-packages/secretstorage/util.py:19: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead Aug 30 01:41:45.821626 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: from cryptography.utils import int_from_bytes Aug 30 01:41:45.821762 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: /usr/lib/python3/dist-packages/secretstorage/util.py:19: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead Aug 30 01:41:45.821762 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: from cryptography.utils import int_from_bytes Aug 30 01:41:45.873450 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] ******************************************************************************** {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2589}} Aug 30 01:41:45.873656 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] Configuration options gathered from: {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2590}} Aug 30 01:41:45.873838 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] command line args: [] {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2591}} Aug 30 01:41:45.874106 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] config files: ['/etc/keystone/keystone.conf'] {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2592}} Aug 30 01:41:45.874284 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] ================================================================================ {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2594}} Aug 30 01:41:45.874539 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] admin_token = **** {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 30 01:41:45.875064 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] config_dir = [] {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 30 01:41:45.875256 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] config_file = ['/etc/keystone/keystone.conf'] {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 30 01:41:45.875437 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] config_source = [] {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 30 01:41:45.875616 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] debug = True {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 30 01:41:45.875815 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] default_log_levels = ['amqp=WARN', 'amqplib=WARN', 'boto=WARN', 'qpid=WARN', 'sqlalchemy=WARN', 'suds=INFO', 'oslo.messaging=INFO', 'oslo_messaging=INFO', 'iso8601=WARN', 'requests.packages.urllib3.connectionpool=WARN', 'urllib3.connectionpool=WARN', 'websocket=WARN', 'requests.packages.urllib3.util.retry=WARN', 'urllib3.util.retry=WARN', 'keystonemiddleware=WARN', 'routes.middleware=WARN', 'stevedore=WARN', 'taskflow=WARN', 'keystoneauth=WARN', 'oslo.cache=INFO', 'oslo_policy=INFO', 'dogpile.core.dogpile=INFO', 'dogpile=INFO', 'routes=INFO'] {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 30 01:41:45.877149 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] default_publisher_id = None {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 30 01:41:45.877149 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] insecure_debug = False {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 30 01:41:45.877298 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] ******************************************************************************** {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2589}} Aug 30 01:41:45.877371 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] instance_format = [instance: %(uuid)s] {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 30 01:41:45.877435 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] Configuration options gathered from: {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2590}} Aug 30 01:41:45.877435 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] command line args: [] {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2591}} Aug 30 01:41:45.877725 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] instance_uuid_format = [instance: %(uuid)s] {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 30 01:41:45.877725 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] list_limit = None {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 30 01:41:45.877859 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] config files: ['/etc/keystone/keystone.conf'] {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2592}} Aug 30 01:41:45.877930 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] log_config_append = None {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 30 01:41:45.877994 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] ================================================================================ {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2594}} Aug 30 01:41:45.878056 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] log_date_format = %Y-%m-%d %H:%M:%S {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 30 01:41:45.878121 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] admin_token = **** {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 30 01:41:45.878183 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] log_dir = None {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 30 01:41:45.878244 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] config_dir = [] {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 30 01:41:45.878310 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] log_file = None {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 30 01:41:45.878372 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] config_file = ['/etc/keystone/keystone.conf'] {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 30 01:41:45.878434 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] log_rotate_interval = 1 {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 30 01:41:45.878496 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] config_source = [] {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 30 01:41:45.878560 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] log_rotate_interval_type = days {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 30 01:41:45.878622 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] debug = True {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 30 01:41:45.878689 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] log_rotation_type = none {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 30 01:41:45.878746 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] default_log_levels = ['amqp=WARN', 'amqplib=WARN', 'boto=WARN', 'qpid=WARN', 'sqlalchemy=WARN', 'suds=INFO', 'oslo.messaging=INFO', 'oslo_messaging=INFO', 'iso8601=WARN', 'requests.packages.urllib3.connectionpool=WARN', 'urllib3.connectionpool=WARN', 'websocket=WARN', 'requests.packages.urllib3.util.retry=WARN', 'urllib3.util.retry=WARN', 'keystonemiddleware=WARN', 'routes.middleware=WARN', 'stevedore=WARN', 'taskflow=WARN', 'keystoneauth=WARN', 'oslo.cache=INFO', 'oslo_policy=INFO', 'dogpile.core.dogpile=INFO', 'dogpile=INFO', 'routes=INFO'] {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 30 01:41:45.878746 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] default_publisher_id = None {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 30 01:41:45.878879 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] logging_context_format_string = %(color)s%(levelname)s %(name)s [%(global_request_id)s %(request_id)s %(project_name)s %(user_name)s%(color)s] %(instance)s%(color)s%(message)s {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 30 01:41:45.878938 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] insecure_debug = False {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 30 01:41:45.879001 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] logging_debug_format_suffix = {{(pid=%(process)d) %(funcName)s %(pathname)s:%(lineno)d}} {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 30 01:41:45.879001 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] logging_default_format_string = %(color)s%(levelname)s %(name)s [-%(color)s] %(instance)s%(color)s%(message)s {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 30 01:41:45.879112 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] instance_format = [instance: %(uuid)s] {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 30 01:41:45.879336 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] logging_exception_prefix = ERROR %(name)s %(instance)s {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 30 01:41:45.879418 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] instance_uuid_format = [instance: %(uuid)s] {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 30 01:41:45.879487 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] logging_user_identity_format = %(user)s %(project)s %(domain)s %(system_scope)s %(user_domain)s %(project_domain)s {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 30 01:41:45.879552 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] list_limit = None {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 30 01:41:45.879714 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] log_config_append = None {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 30 01:41:45.879793 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] max_logfile_count = 30 {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 30 01:41:45.879859 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] log_date_format = %Y-%m-%d %H:%M:%S {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 30 01:41:45.879917 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] max_logfile_size_mb = 200 {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 30 01:41:45.879984 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] log_dir = None {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 30 01:41:45.880136 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] max_param_size = 64 {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 30 01:41:45.880221 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] log_file = None {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 30 01:41:45.880368 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] max_project_tree_depth = 5 {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 30 01:41:45.880454 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] log_rotate_interval = 1 {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 30 01:41:45.880610 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] log_rotate_interval_type = days {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 30 01:41:45.880692 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] max_token_size = 16384 {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 30 01:41:45.880852 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] notification_format = cadf {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 30 01:41:45.881118 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] log_rotation_type = none {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 30 01:41:45.881118 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] logging_context_format_string = %(color)s%(levelname)s %(name)s [%(global_request_id)s %(request_id)s %(project_name)s %(user_name)s%(color)s] %(instance)s%(color)s%(message)s {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 30 01:41:45.881356 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] notification_opt_out = ['identity.authenticate.success', 'identity.authenticate.pending', 'identity.authenticate.failed'] {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 30 01:41:45.881566 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] logging_debug_format_suffix = {{(pid=%(process)d) %(funcName)s %(pathname)s:%(lineno)d}} {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 30 01:41:45.881566 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] logging_default_format_string = %(color)s%(levelname)s %(name)s [-%(color)s] %(instance)s%(color)s%(message)s {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 30 01:41:45.881812 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] public_endpoint = http://10.222.0.27/identity {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 30 01:41:45.881934 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] logging_exception_prefix = ERROR %(name)s %(instance)s {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 30 01:41:45.882068 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] publish_errors = False {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 30 01:41:45.882174 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] logging_user_identity_format = %(user)s %(project)s %(domain)s %(system_scope)s %(user_domain)s %(project_domain)s {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 30 01:41:45.882289 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] pydev_debug_host = None {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 30 01:41:45.882529 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] max_logfile_count = 30 {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 30 01:41:45.882673 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] pydev_debug_port = None {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 30 01:41:45.882781 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] max_logfile_size_mb = 200 {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 30 01:41:45.882893 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] rate_limit_burst = 0 {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 30 01:41:45.882997 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] max_param_size = 64 {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 30 01:41:45.883093 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] rate_limit_except_level = CRITICAL {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 30 01:41:45.883187 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] max_project_tree_depth = 5 {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 30 01:41:45.883281 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] rate_limit_interval = 0 {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 30 01:41:45.883375 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] max_token_size = 16384 {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 30 01:41:45.883477 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] standard_threads = False {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 30 01:41:45.883574 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] notification_format = cadf {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 30 01:41:45.883680 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] strict_password_check = False {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 30 01:41:45.883823 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] notification_opt_out = ['identity.authenticate.success', 'identity.authenticate.pending', 'identity.authenticate.failed'] {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 30 01:41:45.883925 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] syslog_log_facility = LOG_USER {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 30 01:41:45.883925 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] use_eventlog = False {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 30 01:41:45.883925 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] use_journal = False {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 30 01:41:45.884260 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] public_endpoint = http://10.222.0.27/identity {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 30 01:41:45.884357 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] use_json = False {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 30 01:41:45.884459 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] publish_errors = False {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 30 01:41:45.884561 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] use_stderr = False {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 30 01:41:45.884561 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] use_syslog = False {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 30 01:41:45.884741 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] pydev_debug_host = None {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 30 01:41:45.884834 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] watch_log_file = False {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 30 01:41:45.884927 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] pydev_debug_port = None {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 30 01:41:45.884927 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] rate_limit_burst = 0 {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 30 01:41:45.885115 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] oslo_concurrency.disable_process_locking = False {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.885215 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] rate_limit_except_level = CRITICAL {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 30 01:41:45.885369 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] oslo_concurrency.lock_path = None {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.891686 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] rate_limit_interval = 0 {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 30 01:41:45.891824 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] profiler.connection_string = messaging:// {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.891922 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] standard_threads = False {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 30 01:41:45.892004 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] profiler.enabled = False {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.892004 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] profiler.es_doc_type = notification {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.892131 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] strict_password_check = False {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 30 01:41:45.892206 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] profiler.es_scroll_size = 10000 {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.892282 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] syslog_log_facility = LOG_USER {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 30 01:41:45.892282 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] use_eventlog = False {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 30 01:41:45.892469 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] profiler.es_scroll_time = 2m {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.892539 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] use_journal = False {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 30 01:41:45.892608 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] profiler.filter_error_trace = False {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.892676 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] use_json = False {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 30 01:41:45.892744 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] profiler.hmac_keys = SECRET_KEY {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.892941 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] use_stderr = False {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 30 01:41:45.892941 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] use_syslog = False {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 30 01:41:45.893087 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] profiler.sentinel_service_name = mymaster {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.893159 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] watch_log_file = False {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 30 01:41:45.893232 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] profiler.socket_timeout = 0.1 {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.893310 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] oslo_concurrency.disable_process_locking = False {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.893382 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] profiler.trace_sqlalchemy = False {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.893544 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] oslo_concurrency.lock_path = None {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.893621 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_buffer_size = 1000 {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.893691 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] profiler.connection_string = messaging:// {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.893765 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_enabled = False {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.893847 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] profiler.enabled = False {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.893918 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_process_name = {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.893987 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] profiler.es_doc_type = notification {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.894056 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_socket_file = /var/tmp/metrics_collector.sock {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.894124 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] profiler.es_scroll_size = 10000 {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.894192 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_thread_stop_timeout = 10 {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.894269 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] profiler.es_scroll_time = 2m {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.894339 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] oslo_middleware.http_basic_auth_user_file = /etc/htpasswd {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.894463 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] profiler.filter_error_trace = False {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.894539 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] audit.namespace = openstack {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.894608 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] profiler.hmac_keys = SECRET_KEY {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.894677 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] oslo_policy.enforce_new_defaults = False {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.894754 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] profiler.sentinel_service_name = mymaster {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.894830 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] oslo_policy.enforce_scope = False {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.894901 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] profiler.socket_timeout = 0.1 {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.894974 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] oslo_policy.policy_default_rule = default {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.895044 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] profiler.trace_sqlalchemy = False {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.895113 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] oslo_policy.policy_dirs = ['policy.d'] {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.895192 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_buffer_size = 1000 {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.895267 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] oslo_policy.policy_file = policy.yaml {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.895337 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_enabled = False {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.895405 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] oslo_policy.remote_content_type = application/x-www-form-urlencoded {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.895480 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_process_name = {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.895557 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] oslo_policy.remote_ssl_ca_crt_file = None {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.895635 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_socket_file = /var/tmp/metrics_collector.sock {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.895709 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] oslo_policy.remote_ssl_client_crt_file = None {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.895779 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_thread_stop_timeout = 10 {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.895847 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] oslo_policy.remote_ssl_client_key_file = None {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.895920 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] oslo_middleware.http_basic_auth_user_file = /etc/htpasswd {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.895989 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] oslo_policy.remote_ssl_verify_server_crt = False {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.896066 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] audit.namespace = openstack {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.896142 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] keystone_authtoken.admin_password = **** {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.896217 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] oslo_policy.enforce_new_defaults = False {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.896291 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] keystone_authtoken.admin_tenant_name = admin {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.896360 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] oslo_policy.enforce_scope = False {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.896434 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] keystone_authtoken.admin_token = **** {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.896580 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] oslo_policy.policy_default_rule = default {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.896653 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] keystone_authtoken.admin_user = None {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.896722 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] oslo_policy.policy_dirs = ['policy.d'] {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.896789 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] keystone_authtoken.auth_admin_prefix = {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.896857 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] oslo_policy.policy_file = policy.yaml {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.896925 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] keystone_authtoken.auth_host = 127.0.0.1 {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.897002 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] oslo_policy.remote_content_type = application/x-www-form-urlencoded {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.897078 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] keystone_authtoken.auth_port = 35357 {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.897148 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] oslo_policy.remote_ssl_ca_crt_file = None {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.897262 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] keystone_authtoken.auth_protocol = https {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.897340 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] oslo_policy.remote_ssl_client_crt_file = None {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.897409 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] keystone_authtoken.auth_section = None {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.897519 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] oslo_policy.remote_ssl_client_key_file = None {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.897595 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] keystone_authtoken.auth_type = None {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.897670 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] oslo_policy.remote_ssl_verify_server_crt = False {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.897740 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] keystone_authtoken.auth_uri = None {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.897809 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] keystone_authtoken.admin_password = **** {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.897877 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] keystone_authtoken.auth_version = None {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.897953 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] keystone_authtoken.admin_tenant_name = admin {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.898023 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] keystone_authtoken.cache = None {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.898177 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] keystone_authtoken.admin_token = **** {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.898250 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] keystone_authtoken.cafile = None {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.898319 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] keystone_authtoken.admin_user = None {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.898400 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] keystone_authtoken.certfile = None {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.898471 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] keystone_authtoken.auth_admin_prefix = {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.898540 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] keystone_authtoken.delay_auth_decision = False {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.898608 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] keystone_authtoken.auth_host = 127.0.0.1 {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.898712 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] keystone_authtoken.enforce_token_bind = permissive {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.898808 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] keystone_authtoken.auth_port = 35357 {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.898910 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] keystone_authtoken.http_connect_timeout = None {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.899006 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] keystone_authtoken.auth_protocol = https {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.899106 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] keystone_authtoken.http_request_max_retries = 3 {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.899207 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] keystone_authtoken.auth_section = None {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.899302 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] keystone_authtoken.identity_uri = None {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.899429 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] keystone_authtoken.auth_type = None {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.899537 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] keystone_authtoken.include_service_catalog = True {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.899634 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] keystone_authtoken.auth_uri = None {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.899729 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] keystone_authtoken.insecure = False {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.899830 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] keystone_authtoken.auth_version = None {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.899989 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] keystone_authtoken.interface = internal {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.900073 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] keystone_authtoken.cache = None {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.900155 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] keystone_authtoken.keyfile = None {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.900233 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] keystone_authtoken.cafile = None {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.900304 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_conn_get_timeout = 10 {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.900421 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] keystone_authtoken.certfile = None {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.900495 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_dead_retry = 300 {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.900616 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] keystone_authtoken.delay_auth_decision = False {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.900699 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_maxsize = 10 {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.900779 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] keystone_authtoken.enforce_token_bind = permissive {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.900851 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_socket_timeout = 3 {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.900979 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] keystone_authtoken.http_connect_timeout = None {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.901106 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_unused_timeout = 60 {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.901184 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] keystone_authtoken.http_request_max_retries = 3 {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.901255 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] keystone_authtoken.memcache_secret_key = **** {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.901323 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] keystone_authtoken.identity_uri = None {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.901383 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] keystone_authtoken.memcache_security_strategy = None {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.901441 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] keystone_authtoken.include_service_catalog = True {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.901570 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] keystone_authtoken.memcache_use_advanced_pool = True {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.901638 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] keystone_authtoken.insecure = False {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.901696 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] keystone_authtoken.memcached_servers = None {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.901766 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] keystone_authtoken.interface = internal {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.901870 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] keystone_authtoken.region_name = None {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.901948 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] keystone_authtoken.keyfile = None {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.902072 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] keystone_authtoken.service_token_roles = ['service'] {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.902154 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_conn_get_timeout = 10 {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.902226 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] keystone_authtoken.service_token_roles_required = False {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.902339 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_dead_retry = 300 {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.902423 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] keystone_authtoken.service_type = None {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.902494 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_maxsize = 10 {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.902607 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] keystone_authtoken.token_cache_time = 300 {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.902688 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_socket_timeout = 3 {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.902760 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] keystone_authtoken.www_authenticate_uri = None {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.902874 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_unused_timeout = 60 {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.902957 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] cache.backend = dogpile.cache.memcached {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.903029 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] keystone_authtoken.memcache_secret_key = **** {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.903142 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] cache.backend_argument = **** {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.903217 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] keystone_authtoken.memcache_security_strategy = None {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.903329 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] cache.config_prefix = cache.oslo {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.903412 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] keystone_authtoken.memcache_use_advanced_pool = True {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.903514 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] cache.dead_timeout = 60.0 {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.903626 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] keystone_authtoken.memcached_servers = None {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.903707 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] cache.debug_cache_backend = False {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.903821 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] keystone_authtoken.region_name = None {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.903905 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] cache.enable_retry_client = False {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.903977 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] keystone_authtoken.service_token_roles = ['service'] {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.904100 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] cache.enable_socket_keepalive = False {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.904181 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] keystone_authtoken.service_token_roles_required = False {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.904254 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] cache.enabled = True {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.904367 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] keystone_authtoken.service_type = None {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.904441 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] cache.expiration_time = 600 {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.904555 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] keystone_authtoken.token_cache_time = 300 {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.904638 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] cache.hashclient_retry_attempts = 2 {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.904711 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] keystone_authtoken.www_authenticate_uri = None {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.904780 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] cache.hashclient_retry_delay = 1.0 {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.904914 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] cache.backend = dogpile.cache.memcached {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.904974 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] cache.memcache_dead_retry = 300 {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.905076 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] cache.backend_argument = **** {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.905245 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] cache.memcache_password = {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.905326 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] cache.config_prefix = cache.oslo {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.905404 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] cache.memcache_pool_connection_get_timeout = 10 {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.905501 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] cache.dead_timeout = 60.0 {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.905576 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] cache.memcache_pool_flush_on_reconnect = False {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.905647 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] cache.debug_cache_backend = False {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.905729 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] cache.memcache_pool_maxsize = 10 {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.905801 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] cache.enable_retry_client = False {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.905870 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] cache.memcache_pool_unused_timeout = 60 {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.905945 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] cache.enable_socket_keepalive = False {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.906062 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] cache.memcache_sasl_enabled = False {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.906143 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] cache.enabled = True {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.906227 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] cache.memcache_servers = ['localhost:11211'] {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.906313 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] cache.expiration_time = 600 {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.906384 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] cache.memcache_socket_timeout = 1.0 {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.906460 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] cache.hashclient_retry_attempts = 2 {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.906531 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] cache.memcache_username = {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.906606 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] cache.hashclient_retry_delay = 1.0 {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.906678 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] cache.proxies = [] {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.906755 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] cache.memcache_dead_retry = 300 {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.906832 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] cache.retry_attempts = 2 {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.906903 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] cache.memcache_password = {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.906972 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] cache.retry_delay = 0.0 {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.907042 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] cache.memcache_pool_connection_get_timeout = 10 {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.907111 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] cache.socket_keepalive_count = 1 {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.907189 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] cache.memcache_pool_flush_on_reconnect = False {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.907260 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] cache.socket_keepalive_idle = 1 {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.907381 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] cache.memcache_pool_maxsize = 10 {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.907458 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] cache.socket_keepalive_interval = 1 {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.907528 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] cache.memcache_pool_unused_timeout = 60 {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.907597 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] cache.tls_allowed_ciphers = None {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.907674 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] cache.memcache_sasl_enabled = False {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.907746 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] cache.tls_cafile = None {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.907821 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] cache.memcache_servers = ['localhost:11211'] {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.907892 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] cache.tls_certfile = None {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.907962 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] cache.memcache_socket_timeout = 1.0 {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.908032 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] cache.tls_enabled = False {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.908109 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] cache.memcache_username = {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.908186 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] cache.tls_keyfile = None {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.908262 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] cache.proxies = [] {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.908333 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] application_credential.cache_time = None {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.908402 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] cache.retry_attempts = 2 {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.908471 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] application_credential.caching = True {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.908548 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] cache.retry_delay = 0.0 {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.908620 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] application_credential.driver = sql {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.908695 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] cache.socket_keepalive_count = 1 {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.908766 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] application_credential.user_limit = -1 {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.908835 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] cache.socket_keepalive_idle = 1 {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.908910 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] assignment.driver = sql {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.908989 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] cache.socket_keepalive_interval = 1 {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.909060 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] assignment.prohibited_implied_role = ['admin'] {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.909130 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] cache.tls_allowed_ciphers = None {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.909199 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] auth.application_credential = None {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.909268 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] cache.tls_cafile = None {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.909337 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] auth.external = None {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.909413 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] cache.tls_certfile = None {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.909539 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] auth.mapped = None {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.909614 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] cache.tls_enabled = False {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.909818 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] auth.methods = ['external', 'password', 'token', 'oauth1', 'mapped', 'application_credential'] {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.909899 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] cache.tls_keyfile = None {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.910052 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] auth.oauth1 = None {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.910143 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] application_credential.cache_time = None {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.910283 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] auth.password = None {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.910391 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] application_credential.caching = True {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.910516 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] auth.token = None {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.910595 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] application_credential.driver = sql {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.910751 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] application_credential.user_limit = -1 {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.910828 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] catalog.cache_time = None {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.910986 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] catalog.caching = True {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.911071 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] assignment.driver = sql {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.911234 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] catalog.driver = sql {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.911313 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] assignment.prohibited_implied_role = ['admin'] {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.911467 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] catalog.list_limit = None {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.911548 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] auth.application_credential = None {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.911788 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] auth.external = None {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.911882 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] catalog.template_file = default_catalog.templates {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.911958 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] auth.mapped = None {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.912029 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] credential.auth_ttl = 15 {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.912174 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] credential.cache_time = None {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.912252 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] auth.methods = ['external', 'password', 'token', 'oauth1', 'mapped', 'application_credential'] {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.912410 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] auth.oauth1 = None {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.912488 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] credential.caching = True {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.912641 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] auth.password = None {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.912719 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] credential.driver = sql {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.912871 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] auth.token = None {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.912949 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] credential.key_repository = /etc/keystone/credential-keys/ {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.913111 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] catalog.cache_time = None {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.913190 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] credential.provider = fernet {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.913345 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] catalog.caching = True {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.913424 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] credential.user_limit = -1 {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.913613 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] catalog.driver = sql {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.913693 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] domain_config.cache_time = 300 {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.913844 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] catalog.list_limit = None {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.913924 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] domain_config.caching = True {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.914090 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] domain_config.driver = sql {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.914166 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] catalog.template_file = default_catalog.templates {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.914335 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] credential.auth_ttl = 15 {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.914418 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] endpoint_filter.driver = sql {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.914570 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] endpoint_filter.return_all_endpoints_if_no_filter = True {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.914648 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] credential.cache_time = None {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.914803 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] credential.caching = True {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.914880 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] endpoint_policy.driver = sql {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.915047 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] credential.driver = sql {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.915124 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] eventlet_server.admin_bind_host = 0.0.0.0 {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.915293 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] credential.key_repository = /etc/keystone/credential-keys/ {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.915372 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] eventlet_server.admin_port = 35357 {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.915537 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] credential.provider = fernet {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.915615 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] eventlet_server.public_bind_host = 0.0.0.0 {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.915787 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] credential.user_limit = -1 {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.915864 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] eventlet_server.public_port = 5000 {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.916028 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] domain_config.cache_time = 300 {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.916108 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] federation.assertion_prefix = {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.916261 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] domain_config.caching = True {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.916338 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] federation.caching = True {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.916498 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] federation.default_authorization_ttl = 0 {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.916576 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] domain_config.driver = sql {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.917275 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] federation.driver = sql {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.917363 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] endpoint_filter.driver = sql {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.917530 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] federation.federated_domain_name = Federated {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.917649 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] endpoint_filter.return_all_endpoints_if_no_filter = True {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.917759 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] federation.remote_id_attribute = None {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.918007 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] federation.sso_callback_template = /etc/keystone/sso_callback_template.html {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.918152 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] endpoint_policy.driver = sql {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.918237 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] federation.trusted_dashboard = [] {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.918485 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] fernet_receipts.key_repository = /etc/keystone/fernet-keys/ {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.918605 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] eventlet_server.admin_bind_host = 0.0.0.0 {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.918719 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] fernet_receipts.max_active_keys = 3 {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.918832 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] eventlet_server.admin_port = 35357 {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.918957 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] fernet_tokens.key_repository = /etc/keystone/fernet-keys/ {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.919073 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] eventlet_server.public_bind_host = 0.0.0.0 {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.919186 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] fernet_tokens.max_active_keys = 3 {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.919300 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] eventlet_server.public_port = 5000 {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.919413 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] identity.cache_time = 600 {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.919527 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] federation.assertion_prefix = {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.919650 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] identity.caching = True {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.919765 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] federation.caching = True {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.919878 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] identity.default_domain_id = default {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.919991 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] federation.default_authorization_ttl = 0 {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.920111 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] identity.domain_config_dir = /etc/keystone/domains {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.920226 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] federation.driver = sql {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.920347 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] identity.domain_configurations_from_database = False {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.920462 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] federation.federated_domain_name = Federated {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.920575 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] identity.domain_specific_drivers_enabled = False {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.920690 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] federation.remote_id_attribute = None {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.920802 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] identity.driver = sql {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.920915 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] federation.sso_callback_template = /etc/keystone/sso_callback_template.html {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.921038 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] identity.list_limit = None {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.921152 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] federation.trusted_dashboard = [] {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.921272 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] identity.max_password_length = 4096 {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.921386 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] fernet_receipts.key_repository = /etc/keystone/fernet-keys/ {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.921526 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] identity.password_hash_algorithm = bcrypt {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.921642 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] fernet_receipts.max_active_keys = 3 {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.921769 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] identity.password_hash_rounds = 4 {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.921885 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] fernet_tokens.key_repository = /etc/keystone/fernet-keys/ {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.921999 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] identity.salt_bytesize = None {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.922119 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] fernet_tokens.max_active_keys = 3 {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.922300 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] identity.scrypt_block_size = None {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.922432 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] identity.cache_time = 600 {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.922544 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] identity.scrypt_parallelism = None {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.922644 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] identity.caching = True {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.922763 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] identity_mapping.backward_compatible_ids = True {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.922878 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] identity.default_domain_id = default {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.922990 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] identity_mapping.driver = sql {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.923102 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] identity.domain_config_dir = /etc/keystone/domains {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.923224 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] identity_mapping.generator = sha256 {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.923338 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] identity.domain_configurations_from_database = False {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.923450 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] jwt_tokens.jws_private_key_repository = /etc/keystone/jws-keys/private {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.923563 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] identity.domain_specific_drivers_enabled = False {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.923675 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] jwt_tokens.jws_public_key_repository = /etc/keystone/jws-keys/public {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.923801 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] identity.driver = sql {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.923916 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] ldap.alias_dereferencing = default {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.924029 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] identity.list_limit = None {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.924141 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] ldap.auth_pool_connection_lifetime = 60 {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.924254 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] identity.max_password_length = 4096 {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.924366 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] ldap.auth_pool_size = 100 {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.924487 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] identity.password_hash_algorithm = bcrypt {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.924602 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] ldap.chase_referrals = None {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.924715 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] identity.password_hash_rounds = 4 {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.924828 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] ldap.connection_timeout = -1 {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.924932 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] identity.salt_bytesize = None {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.925043 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] ldap.debug_level = None {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.925164 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] identity.scrypt_block_size = None {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.925278 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] ldap.group_ad_nesting = False {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.925391 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] identity.scrypt_parallelism = None {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.925550 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] ldap.group_additional_attribute_mapping = [] {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.925675 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] identity_mapping.backward_compatible_ids = True {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.925790 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] ldap.group_attribute_ignore = [] {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.925914 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] identity_mapping.driver = sql {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.926028 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] ldap.group_desc_attribute = description {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.926244 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] identity_mapping.generator = sha256 {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.926363 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] ldap.group_filter = None {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.926476 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] jwt_tokens.jws_private_key_repository = /etc/keystone/jws-keys/private {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.926589 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] ldap.group_id_attribute = cn {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.926710 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] jwt_tokens.jws_public_key_repository = /etc/keystone/jws-keys/public {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.926825 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] ldap.group_member_attribute = member {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.926938 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] ldap.alias_dereferencing = default {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.927051 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] ldap.group_members_are_ids = False {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.927164 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] ldap.auth_pool_connection_lifetime = 60 {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.927276 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] ldap.group_name_attribute = ou {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.927397 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] ldap.auth_pool_size = 100 {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.927511 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] ldap.group_objectclass = groupOfNames {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.927624 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] ldap.chase_referrals = None {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.927736 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] ldap.group_tree_dn = None {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.927736 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] ldap.page_size = 0 {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.927949 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] ldap.connection_timeout = -1 {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.928071 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] ldap.password = **** {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.928186 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] ldap.debug_level = None {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.928299 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] ldap.pool_connection_lifetime = 600 {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.928411 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] ldap.group_ad_nesting = False {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.928524 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] ldap.pool_connection_timeout = -1 {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.928643 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] ldap.group_additional_attribute_mapping = [] {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.928765 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] ldap.pool_retry_delay = 0.1 {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.928879 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] ldap.group_attribute_ignore = [] {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.929024 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] ldap.pool_retry_max = 3 {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.929122 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] ldap.group_desc_attribute = description {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.929217 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] ldap.pool_size = 10 {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.929313 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] ldap.group_filter = None {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.929417 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] ldap.query_scope = one {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.929548 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] ldap.group_id_attribute = cn {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.929647 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] ldap.suffix = cn=example,cn=com {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.929743 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] ldap.group_member_attribute = member {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.929838 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] ldap.tls_cacertdir = None {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.929933 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] ldap.group_members_are_ids = False {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.934592 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] ldap.tls_cacertfile = None {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.934736 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] ldap.group_name_attribute = ou {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.934819 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] ldap.tls_req_cert = demand {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.934930 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] ldap.group_objectclass = groupOfNames {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.935009 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] ldap.url = ldap://localhost {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.935114 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] ldap.group_tree_dn = None {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.935207 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] ldap.use_auth_pool = True {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.935327 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] ldap.page_size = 0 {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.935403 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] ldap.use_pool = True {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.935519 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] ldap.password = **** {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.935595 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] ldap.use_tls = False {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.935661 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] ldap.pool_connection_lifetime = 600 {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.935740 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] ldap.user = None {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.935813 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] ldap.pool_connection_timeout = -1 {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.935884 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] ldap.user_additional_attribute_mapping = [] {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.935954 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] ldap.pool_retry_delay = 0.1 {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.935954 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] ldap.pool_retry_max = 3 {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.936082 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] ldap.user_attribute_ignore = ['default_project_id'] {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.936082 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] ldap.user_default_project_id_attribute = None {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.936219 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] ldap.pool_size = 10 {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.936289 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] ldap.user_description_attribute = description {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.936359 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] ldap.query_scope = one {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.936428 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] ldap.user_enabled_attribute = enabled {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.936498 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] ldap.suffix = cn=example,cn=com {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.936576 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] ldap.user_enabled_default = True {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.936576 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] ldap.user_enabled_emulation = False {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.936754 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] ldap.tls_cacertdir = None {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.936754 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] ldap.tls_cacertfile = None {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.936930 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] ldap.user_enabled_emulation_dn = None {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.936930 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] ldap.user_enabled_emulation_use_group_config = False {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.937112 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] ldap.tls_req_cert = demand {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.937180 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] ldap.user_enabled_invert = False {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.937251 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] ldap.url = ldap://localhost {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.937321 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] ldap.user_enabled_mask = 0 {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.937398 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] ldap.use_auth_pool = True {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.937398 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] ldap.use_pool = True {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.937556 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] ldap.user_filter = None {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.937630 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] ldap.use_tls = False {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.937699 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] ldap.user_id_attribute = cn {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.937776 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] ldap.user = None {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.937847 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] ldap.user_mail_attribute = mail {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.937917 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] ldap.user_additional_attribute_mapping = [] {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.938043 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] ldap.user_name_attribute = sn {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.938120 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] ldap.user_attribute_ignore = ['default_project_id'] {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.938190 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] ldap.user_objectclass = inetOrgPerson {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.938261 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] ldap.user_default_project_id_attribute = None {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.938330 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] ldap.user_pass_attribute = userPassword {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.938408 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] ldap.user_description_attribute = description {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.938481 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] ldap.user_tree_dn = None {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.938552 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] ldap.user_enabled_attribute = enabled {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.938667 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] memcache.dead_retry = 300 {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.938742 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] ldap.user_enabled_default = True {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.938896 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] memcache.pool_connection_get_timeout = 10 {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.938985 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] ldap.user_enabled_emulation = False {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.939104 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] memcache.pool_maxsize = 10 {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.939180 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] ldap.user_enabled_emulation_dn = None {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.939293 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] memcache.pool_unused_timeout = 60 {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.939370 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] ldap.user_enabled_emulation_use_group_config = False {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.939483 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] memcache.socket_timeout = 3 {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.939568 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] ldap.user_enabled_invert = False {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.939684 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] oauth1.access_token_duration = 86400 {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.939761 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] ldap.user_enabled_mask = 0 {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.939875 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] oauth1.driver = sql {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.939950 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] ldap.user_filter = None {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.940064 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] oauth1.request_token_duration = 28800 {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.940148 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] ldap.user_id_attribute = cn {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.940221 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] policy.driver = sql {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.940292 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] ldap.user_mail_attribute = mail {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.940364 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] policy.list_limit = None {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.940436 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] ldap.user_name_attribute = sn {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.940550 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] receipt.cache_on_issue = True {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.940635 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] ldap.user_objectclass = inetOrgPerson {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.940788 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] receipt.cache_time = 300 {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.940871 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] ldap.user_pass_attribute = userPassword {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.940979 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] receipt.caching = True {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.941058 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] ldap.user_tree_dn = None {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.941173 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] receipt.expiration = 300 {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.941257 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] memcache.dead_retry = 300 {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.941257 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] memcache.pool_connection_get_timeout = 10 {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.941396 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] receipt.provider = fernet {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.941499 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] memcache.pool_maxsize = 10 {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.941575 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] resource.admin_project_domain_name = None {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.941575 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] resource.admin_project_name = None {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.941782 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] memcache.pool_unused_timeout = 60 {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.941858 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] resource.cache_time = None {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.941928 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] memcache.socket_timeout = 3 {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.942004 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] resource.caching = True {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.942082 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] oauth1.access_token_duration = 86400 {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.942082 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] oauth1.driver = sql {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.942219 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] resource.domain_name_url_safe = off {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.942298 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] oauth1.request_token_duration = 28800 {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.942369 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] resource.driver = sql {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.942440 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] policy.driver = sql {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.942509 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] resource.list_limit = None {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.942579 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] policy.list_limit = None {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.942657 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] resource.project_name_url_safe = off {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.942729 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] receipt.cache_on_issue = True {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.942799 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] revoke.cache_time = 3600 {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.942869 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] receipt.cache_time = 300 {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.942939 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] revoke.caching = True {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.943009 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] receipt.caching = True {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.943091 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] revoke.driver = sql {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.943164 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] receipt.expiration = 300 {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.943235 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] revoke.expiration_buffer = 1800 {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.943305 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] receipt.provider = fernet {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.943375 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] role.cache_time = None {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.943445 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] resource.admin_project_domain_name = None {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.943523 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] role.caching = True {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.943596 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] resource.admin_project_name = None {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.943666 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] role.driver = sql {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.943737 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] resource.cache_time = None {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.943796 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] role.list_limit = None {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.943865 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] resource.caching = True {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.943942 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] saml.assertion_expiration_time = 3600 {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.944014 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] resource.domain_name_url_safe = off {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.944154 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] saml.certfile = /etc/keystone/ssl/certs/signing_cert.pem {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.944235 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] resource.driver = sql {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.944403 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] saml.idp_contact_company = Example, Inc. {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.944610 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] resource.list_limit = None {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.944695 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] saml.idp_contact_email = support@example.com {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.944770 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] resource.project_name_url_safe = off {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.944885 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] saml.idp_contact_name = SAML Identity Provider Support {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.944966 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] revoke.cache_time = 3600 {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.945118 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] revoke.caching = True {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.945197 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] saml.idp_contact_surname = Support {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.945365 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] revoke.driver = sql {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.945444 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] saml.idp_contact_telephone = +1 800 555 0100 {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.945620 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] revoke.expiration_buffer = 1800 {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.945696 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] saml.idp_contact_type = other {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.945854 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] role.cache_time = None {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.945933 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] saml.idp_entity_id = None {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.946089 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] role.caching = True {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.946169 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] saml.idp_lang = en {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.946334 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] role.driver = sql {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.946413 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] saml.idp_metadata_path = /etc/keystone/saml2_idp_metadata.xml {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.946566 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] role.list_limit = None {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.946645 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] saml.idp_organization_display_name = OpenStack SAML Identity Provider {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.946814 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] saml.idp_organization_name = SAML Identity Provider {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.946894 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] saml.assertion_expiration_time = 3600 {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.947059 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] saml.certfile = /etc/keystone/ssl/certs/signing_cert.pem {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.947247 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] saml.idp_organization_url = https://example.com/ {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.947326 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] saml.idp_contact_company = Example, Inc. {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.947493 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] saml.idp_sso_endpoint = None {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.947573 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] saml.idp_contact_email = support@example.com {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.947748 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] saml.keyfile = /etc/keystone/ssl/private/signing_key.pem {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.947827 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] saml.idp_contact_name = SAML Identity Provider Support {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.948070 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] saml.relay_state_prefix = ss:mem: {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.948156 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] saml.idp_contact_surname = Support {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.948381 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] saml.idp_contact_telephone = +1 800 555 0100 {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.948500 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] saml.xmlsec1_binary = xmlsec1 {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.948615 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] saml.idp_contact_type = other {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.948857 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] saml.idp_entity_id = None {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.949025 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] security_compliance.change_password_upon_first_use = False {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.949200 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] security_compliance.disable_user_account_days_inactive = None {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.949333 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] saml.idp_lang = en {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.949333 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] saml.idp_metadata_path = /etc/keystone/saml2_idp_metadata.xml {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.949585 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] security_compliance.lockout_duration = 10 {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.949700 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] saml.idp_organization_display_name = OpenStack SAML Identity Provider {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.949814 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] security_compliance.lockout_failure_attempts = 2 {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.949927 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] saml.idp_organization_name = SAML Identity Provider {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.950056 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] security_compliance.minimum_password_age = 0 {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.950056 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] security_compliance.password_expires_days = None {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.950271 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] saml.idp_organization_url = https://example.com/ {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.950384 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] security_compliance.password_regex = None {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.950498 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] saml.idp_sso_endpoint = None {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.950611 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] security_compliance.password_regex_description = None {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.950732 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] saml.keyfile = /etc/keystone/ssl/private/signing_key.pem {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.950847 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] security_compliance.unique_last_password_count = 2 {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.950961 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] saml.relay_state_prefix = ss:mem: {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.951075 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] shadow_users.driver = sql {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.951195 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] saml.xmlsec1_binary = xmlsec1 {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.951309 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] token.allow_expired_window = 172800 {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.951431 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] security_compliance.change_password_upon_first_use = False {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.951545 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] token.allow_rescope_scoped_token = True {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.951659 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] security_compliance.disable_user_account_days_inactive = None {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.951772 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] token.cache_on_issue = True {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.951885 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] security_compliance.lockout_duration = 10 {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.951998 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] token.cache_time = None {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.952120 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] security_compliance.lockout_failure_attempts = 2 {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.952235 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] token.caching = True {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.952347 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] security_compliance.minimum_password_age = 0 {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.952460 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] token.expiration = 3600 {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.952573 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] security_compliance.password_expires_days = None {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.952687 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] token.provider = fernet {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.952809 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] security_compliance.password_regex = None {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.952923 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] token.revoke_by_id = True {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.953037 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] security_compliance.password_regex_description = None {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.953150 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] tokenless_auth.issuer_attribute = SSL_CLIENT_I_DN {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.953271 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] security_compliance.unique_last_password_count = 2 {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.953393 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] tokenless_auth.protocol = x509 {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.953521 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] shadow_users.driver = sql {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.953636 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] tokenless_auth.trusted_issuer = [] {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.953750 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] token.allow_expired_window = 172800 {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.953863 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] totp.included_previous_windows = 1 {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.953976 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] token.allow_rescope_scoped_token = True {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.954100 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] trust.allow_redelegation = False {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.954217 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] token.cache_on_issue = True {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.954329 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] trust.driver = sql {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.954442 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] token.cache_time = None {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.954554 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] trust.max_redelegation_count = 3 {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.954667 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] token.caching = True {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.954788 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] unified_limit.cache_time = None {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.954903 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] token.expiration = 3600 {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.955016 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] unified_limit.caching = True {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.955129 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] token.provider = fernet {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.955242 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] unified_limit.driver = sql {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.955356 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] token.revoke_by_id = True {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.955477 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] unified_limit.enforcement_model = flat {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.955591 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] tokenless_auth.issuer_attribute = SSL_CLIENT_I_DN {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.955705 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] unified_limit.list_limit = None {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.955827 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] tokenless_auth.protocol = x509 {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.955949 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] wsgi.debug_middleware = False {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.956065 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] tokenless_auth.trusted_issuer = [] {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.956178 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] database.backend = sqlalchemy {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.956396 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] totp.included_previous_windows = 1 {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.956575 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] database.connection = **** {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.956694 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] trust.allow_redelegation = False {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.956807 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] database.connection_debug = 0 {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.956921 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] trust.driver = sql {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.957034 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] database.connection_parameters = {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.957147 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] trust.max_redelegation_count = 3 {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.957270 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] database.connection_recycle_time = 3600 {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.957387 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] unified_limit.cache_time = None {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.957577 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] database.connection_trace = False {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.957694 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] unified_limit.caching = True {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.957808 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] database.db_inc_retry_interval = True {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.957921 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] unified_limit.driver = sql {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.958048 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] database.db_max_retries = 20 {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.958164 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] unified_limit.enforcement_model = flat {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.958277 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] database.db_max_retry_interval = 10 {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.958391 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] unified_limit.list_limit = None {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.958517 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] database.db_retry_interval = 1 {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.958641 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] wsgi.debug_middleware = False {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.958756 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] database.max_overflow = 50 {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.958879 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] database.backend = sqlalchemy {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.958994 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] database.max_pool_size = 5 {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.959107 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] database.connection = **** {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.959225 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] database.max_retries = 10 {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.959338 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] database.connection_debug = 0 {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.959452 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] database.mysql_enable_ndb = False {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.959641 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] database.connection_parameters = {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.959758 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] database.mysql_sql_mode = TRADITIONAL {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.959872 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] database.connection_recycle_time = 3600 {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.959985 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] database.pool_timeout = None {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.960097 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] database.connection_trace = False {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.960211 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] database.retry_interval = 10 {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.960332 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] database.db_inc_retry_interval = True {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.960447 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] database.slave_connection = **** {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.960561 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] database.db_max_retries = 20 {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.960675 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] database.sqlite_synchronous = True {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.960788 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] database.db_max_retry_interval = 10 {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.960901 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] database.use_db_reconnect = False {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.961023 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] database.db_retry_interval = 1 {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.961138 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] ******************************************************************************** {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2613}} Aug 30 01:41:45.961251 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] database.max_overflow = 50 {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.961251 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] database.max_pool_size = 5 {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.961251 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] database.max_retries = 10 {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.961584 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] database.mysql_enable_ndb = False {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.961742 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] database.mysql_sql_mode = TRADITIONAL {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.961977 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] database.pool_timeout = None {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.962222 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] database.retry_interval = 10 {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.962450 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] database.slave_connection = **** {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.962689 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] database.sqlite_synchronous = True {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.962922 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] database.use_db_reconnect = False {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 30 01:41:45.963117 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG uwsgi [-] ******************************************************************************** {{(pid=81578) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2613}} Aug 30 01:41:45.991749 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_user_delete_callback` subscribed to event `identity.user.deleted`. {{(pid=81577) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 30 01:41:45.991907 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_user_delete_callback` subscribed to event `identity.user.disabled`. {{(pid=81577) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 30 01:41:45.992021 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_assignment_removal` subscribed to event `identity.remove_application_credentials_for_user.internal`. {{(pid=81577) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 30 01:41:45.992991 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_user_delete_callback` subscribed to event `identity.user.deleted`. {{(pid=81578) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 30 01:41:45.993222 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_user_delete_callback` subscribed to event `identity.user.disabled`. {{(pid=81578) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 30 01:41:45.993448 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_assignment_removal` subscribed to event `identity.remove_application_credentials_for_user.internal`. {{(pid=81578) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 30 01:41:46.000710 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [-] Callback: `keystone.assignment.core.Manager._delete_domain_assignments` subscribed to event `identity.domain.deleted`. {{(pid=81577) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 30 01:41:46.001844 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [-] Callback: `keystone.assignment.core.Manager._delete_domain_assignments` subscribed to event `identity.domain.deleted`. {{(pid=81578) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 30 01:41:46.029135 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [-] Callback: `keystone.catalog.core.Manager._on_project_or_endpoint_delete` subscribed to event `identity.project.deleted`. {{(pid=81577) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 30 01:41:46.032346 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [-] Callback: `keystone.catalog.core.Manager._on_project_or_endpoint_delete` subscribed to event `identity.endpoint.deleted`. {{(pid=81577) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 30 01:41:46.032447 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [-] Callback: `keystone.catalog.core.Manager._on_project_or_endpoint_delete` subscribed to event `identity.project.deleted`. {{(pid=81578) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 30 01:41:46.032447 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [-] Callback: `keystone.catalog.core.Manager._on_project_or_endpoint_delete` subscribed to event `identity.endpoint.deleted`. {{(pid=81578) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 30 01:41:46.094508 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [-] Callback: `keystone.federation.core.Manager._cleanup_identity_provider` subscribed to event `identity.domain_deleted.internal`. {{(pid=81577) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 30 01:41:46.094693 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [-] Callback: `keystone.federation.core.Manager._cleanup_identity_provider` subscribed to event `identity.domain_deleted.internal`. {{(pid=81578) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 30 01:41:46.143435 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [-] Callback: `keystone.identity.core.Manager._domain_deleted` subscribed to event `identity.domain_deleted.internal`. {{(pid=81577) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 30 01:41:46.143670 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [-] Callback: `keystone.identity.core.Manager._unset_default_project` subscribed to event `identity.project.deleted`. {{(pid=81577) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 30 01:41:46.143898 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [-] Callback: `keystone.identity.core.Manager._domain_deleted` subscribed to event `identity.domain_deleted.internal`. {{(pid=81578) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 30 01:41:46.144136 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [-] Callback: `keystone.identity.core.Manager._unset_default_project` subscribed to event `identity.project.deleted`. {{(pid=81578) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 30 01:41:46.302817 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._trust_callback` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=81577) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 30 01:41:46.303048 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._consumer_callback` subscribed to event `identity.OS-OAUTH1:consumer.deleted`. {{(pid=81577) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 30 01:41:46.303274 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.user.deleted`. {{(pid=81577) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 30 01:41:46.303494 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._project_callback` subscribed to event `identity.project.deleted`. {{(pid=81577) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 30 01:41:46.303710 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.user.disabled`. {{(pid=81577) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 30 01:41:46.303926 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.persist_revocation_event_for_user.internal`. {{(pid=81577) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 30 01:41:46.305871 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._trust_callback` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=81578) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 30 01:41:46.306101 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._consumer_callback` subscribed to event `identity.OS-OAUTH1:consumer.deleted`. {{(pid=81578) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 30 01:41:46.306327 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.user.deleted`. {{(pid=81578) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 30 01:41:46.306546 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._project_callback` subscribed to event `identity.project.deleted`. {{(pid=81578) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 30 01:41:46.306791 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.user.disabled`. {{(pid=81578) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 30 01:41:46.307008 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.persist_revocation_event_for_user.internal`. {{(pid=81578) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 30 01:41:46.328028 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=81577) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 30 01:41:46.328222 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=81578) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 30 01:41:46.328343 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.user.deleted`. {{(pid=81577) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 30 01:41:46.328464 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.user.deleted`. {{(pid=81578) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 30 01:41:46.328577 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.domain.deleted`. {{(pid=81577) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 30 01:41:46.328656 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.user.disabled`. {{(pid=81577) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 30 01:41:46.328752 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.domain.deleted`. {{(pid=81578) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 30 01:41:46.328873 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.domain.disabled`. {{(pid=81577) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 30 01:41:46.328959 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.user.disabled`. {{(pid=81578) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 30 01:41:46.329088 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.project.disabled`. {{(pid=81577) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 30 01:41:46.329171 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.domain.disabled`. {{(pid=81578) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 30 01:41:46.329305 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.project.disabled`. {{(pid=81578) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 30 01:41:46.329421 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.invalidate_token_cache.internal`. {{(pid=81577) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 30 01:41:46.329595 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.invalidate_token_cache.internal`. {{(pid=81578) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 30 01:41:46.346536 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [-] Callback: `keystone.trust.core.Manager._on_user_delete` subscribed to event `identity.user.deleted`. {{(pid=81577) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 30 01:41:46.347392 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [-] Callback: `keystone.trust.core.Manager._on_user_delete` subscribed to event `identity.user.deleted`. {{(pid=81578) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 30 01:41:46.359733 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=81577) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 30 01:41:46.359967 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.user.deleted`. {{(pid=81577) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 30 01:41:46.360193 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.domain.deleted`. {{(pid=81577) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 30 01:41:46.360418 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.user.disabled`. {{(pid=81577) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 30 01:41:46.360556 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=81578) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 30 01:41:46.360717 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.domain.disabled`. {{(pid=81577) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 30 01:41:46.360817 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.user.deleted`. {{(pid=81578) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 30 01:41:46.360900 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.project.disabled`. {{(pid=81577) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 30 01:41:46.361030 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.domain.deleted`. {{(pid=81578) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 30 01:41:46.361117 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.invalidate_token_cache.internal`. {{(pid=81577) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 30 01:41:46.361254 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.user.disabled`. {{(pid=81578) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 30 01:41:46.361499 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.domain.disabled`. {{(pid=81578) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 30 01:41:46.361721 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.project.disabled`. {{(pid=81578) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 30 01:41:46.361937 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.invalidate_token_cache.internal`. {{(pid=81578) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 30 01:41:46.394946 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/projects' {}] {{(pid=81577) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 01:41:46.395146 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.common [-] Adding additional resource route (alternate) to API auth: ['/OS-FEDERATION/projects' {}] {{(pid=81577) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:433}} Aug 30 01:41:46.395493 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/domains' {}] {{(pid=81577) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 01:41:46.395695 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.common [-] Adding additional resource route (alternate) to API auth: ['/OS-FEDERATION/domains' {}] {{(pid=81577) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:433}} Aug 30 01:41:46.395973 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/system' {}] {{(pid=81577) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 01:41:46.396237 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/catalog' {}] {{(pid=81577) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 01:41:46.396500 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/tokens/OS-PKI/revoked' {}] {{(pid=81577) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 01:41:46.396762 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/tokens' {}] {{(pid=81577) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 01:41:46.396986 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/projects' {}] {{(pid=81578) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 01:41:46.397208 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.common [-] Adding additional resource route (alternate) to API auth: ['/OS-FEDERATION/projects' {}] {{(pid=81578) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:433}} Aug 30 01:41:46.397589 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/domains' {}] {{(pid=81578) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 01:41:46.397817 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.common [-] Adding additional resource route (alternate) to API auth: ['/OS-FEDERATION/domains' {}] {{(pid=81578) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:433}} Aug 30 01:41:46.398096 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/system' {}] {{(pid=81578) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 01:41:46.398363 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/catalog' {}] {{(pid=81578) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 01:41:46.398896 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/tokens/OS-PKI/revoked' {}] {{(pid=81578) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 01:41:46.399163 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/tokens' {}] {{(pid=81578) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 01:41:46.406932 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/saml2' {}] {{(pid=81577) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 01:41:46.407209 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/saml2/ecp' {}] {{(pid=81577) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 01:41:46.407479 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/websso/' {}] {{(pid=81577) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 01:41:46.407786 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/identity_providers//protocols//websso' {}] {{(pid=81577) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 01:41:46.409340 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/saml2' {}] {{(pid=81578) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 01:41:46.409649 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/saml2/ecp' {}] {{(pid=81578) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 01:41:46.409928 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/websso/' {}] {{(pid=81578) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 01:41:46.410214 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/identity_providers//protocols//websso' {}] {{(pid=81578) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 01:41:46.414740 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.common [-] Adding standard routes to API credentials for `type` (API Prefix: ) [/credentials, /credentials/] {{(pid=81577) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 30 01:41:46.417268 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.common [-] Adding standard routes to API credentials for `type` (API Prefix: ) [/credentials, /credentials/] {{(pid=81578) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 30 01:41:46.417674 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.common [-] Adding standard routes to API domains for `type` (API Prefix: ) [/domains, /domains/] {{(pid=81577) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 30 01:41:46.417961 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config' {}] {{(pid=81577) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 01:41:46.418246 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config/' {}] {{(pid=81577) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 01:41:46.418533 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config//' {}] {{(pid=81577) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 01:41:46.418825 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config/default' {}] {{(pid=81577) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 01:41:46.419084 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config//default' {}] {{(pid=81577) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 01:41:46.419360 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config///default' {}] {{(pid=81577) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 01:41:46.419638 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//users//roles' {}] {{(pid=81577) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 01:41:46.419912 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//users//roles/' {}] {{(pid=81577) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 01:41:46.420189 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//groups//roles' {}] {{(pid=81577) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 01:41:46.420282 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.common [-] Adding standard routes to API domains for `type` (API Prefix: ) [/domains, /domains/] {{(pid=81578) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 30 01:41:46.420456 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//groups//roles/' {}] {{(pid=81577) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 01:41:46.420545 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config' {}] {{(pid=81578) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 01:41:46.420833 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config/' {}] {{(pid=81578) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 01:41:46.421124 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config//' {}] {{(pid=81578) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 01:41:46.421411 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config/default' {}] {{(pid=81578) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 01:41:46.421703 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config//default' {}] {{(pid=81578) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 01:41:46.421977 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config///default' {}] {{(pid=81578) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 01:41:46.422254 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//users//roles' {}] {{(pid=81578) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 01:41:46.422644 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//users//roles/' {}] {{(pid=81578) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 01:41:46.422931 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//groups//roles' {}] {{(pid=81578) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 01:41:46.423208 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//groups//roles/' {}] {{(pid=81578) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 01:41:46.443036 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.common [-] Adding resource routes to API ec2tokens: ['/ec2tokens' {}] {{(pid=81577) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 01:41:46.444494 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.common [-] Adding standard routes to API endpoints for `type` (API Prefix: ) [/endpoints, /endpoints/] {{(pid=81577) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 30 01:41:46.444777 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.common [-] Adding resource routes to API endpoints: ['/endpoints//OS-ENDPOINT-POLICY/policy' {}] {{(pid=81577) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 01:41:46.446412 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.common [-] Adding resource routes to API ec2tokens: ['/ec2tokens' {}] {{(pid=81578) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 01:41:46.448522 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.common [-] Adding standard routes to API endpoints for `type` (API Prefix: ) [/endpoints, /endpoints/] {{(pid=81578) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 30 01:41:46.448893 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.common [-] Adding resource routes to API endpoints: ['/endpoints//OS-ENDPOINT-POLICY/policy' {}] {{(pid=81578) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 01:41:46.449905 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.common [-] Adding standard routes to API groups for `type` (API Prefix: ) [/groups, /groups/] {{(pid=81577) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 30 01:41:46.450189 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.common [-] Adding resource routes to API groups: ['/groups//users' {}] {{(pid=81577) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 01:41:46.455259 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.common [-] Adding resource routes to API groups: ['/groups//users/' {}] {{(pid=81577) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 01:41:46.455341 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.common [-] Adding standard routes to API groups for `type` (API Prefix: ) [/groups, /groups/] {{(pid=81578) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 30 01:41:46.455341 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.common [-] Adding resource routes to API groups: ['/groups//users' {}] {{(pid=81578) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 01:41:46.455341 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.common [-] Adding resource routes to API groups: ['/groups//users/' {}] {{(pid=81578) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 01:41:46.456918 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.common [-] Adding standard routes to API limits for `type` (API Prefix: ) [/limits, /limits/] {{(pid=81577) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 30 01:41:46.457203 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.common [-] Adding resource routes to API limits: ['/limits/model' {}] {{(pid=81577) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 01:41:46.460812 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.common [-] Adding standard routes to API limits for `type` (API Prefix: ) [/limits, /limits/] {{(pid=81578) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 30 01:41:46.461103 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.common [-] Adding resource routes to API limits: ['/limits/model' {}] {{(pid=81578) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 01:41:46.461212 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.common [-] Adding standard routes to API OS-EP-FILTER for `type` (API Prefix: /OS-EP-FILTER) [/endpoint_groups, /endpoint_groups/] {{(pid=81577) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 30 01:41:46.461534 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoints//projects' {}] {{(pid=81577) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 01:41:46.461824 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoints/' {}] {{(pid=81577) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 01:41:46.462108 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoints' {}] {{(pid=81577) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 01:41:46.462383 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoint_groups' {}] {{(pid=81577) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 01:41:46.465047 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//endpoints' {}] {{(pid=81577) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 01:41:46.465338 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//projects' {}] {{(pid=81577) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 01:41:46.465646 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//projects/' {}] {{(pid=81577) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 01:41:46.465900 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.common [-] Adding standard routes to API OS-EP-FILTER for `type` (API Prefix: /OS-EP-FILTER) [/endpoint_groups, /endpoint_groups/] {{(pid=81578) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 30 01:41:46.466195 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoints//projects' {}] {{(pid=81578) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 01:41:46.466486 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoints/' {}] {{(pid=81578) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 01:41:46.466818 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoints' {}] {{(pid=81578) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 01:41:46.467101 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoint_groups' {}] {{(pid=81578) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 01:41:46.467380 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//endpoints' {}] {{(pid=81578) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 01:41:46.467653 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//projects' {}] {{(pid=81578) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 01:41:46.467926 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//projects/' {}] {{(pid=81578) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 01:41:46.481752 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-FEDERATION: ['/saml2/metadata' {}] {{(pid=81577) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 01:41:46.482035 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-FEDERATION: ['/identity_providers//protocols//auth' {}] {{(pid=81577) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 01:41:46.485771 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-FEDERATION: ['/saml2/metadata' {}] {{(pid=81578) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 01:41:46.485918 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.common [-] Adding standard routes to API identity_providers for `type` (API Prefix: /OS-FEDERATION) [/identity_providers, /identity_providers/] {{(pid=81577) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 30 01:41:46.486060 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-FEDERATION: ['/identity_providers//protocols//auth' {}] {{(pid=81578) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 01:41:46.489017 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.common [-] Adding resource routes to API protocols: ['/OS-FEDERATION/identity_providers//protocols/' {}] {{(pid=81577) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 01:41:46.489312 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.common [-] Adding resource routes to API protocols: ['/OS-FEDERATION/identity_providers//protocols' {}] {{(pid=81577) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 01:41:46.490091 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.common [-] Adding standard routes to API identity_providers for `type` (API Prefix: /OS-FEDERATION) [/identity_providers, /identity_providers/] {{(pid=81578) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 30 01:41:46.493357 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.common [-] Adding resource routes to API protocols: ['/OS-FEDERATION/identity_providers//protocols/' {}] {{(pid=81578) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 01:41:46.493676 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.common [-] Adding standard routes to API mappings for `type` (API Prefix: /OS-FEDERATION) [/mappings, /mappings/] {{(pid=81577) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 30 01:41:46.493764 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.common [-] Adding resource routes to API protocols: ['/OS-FEDERATION/identity_providers//protocols' {}] {{(pid=81578) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 01:41:46.496891 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.common [-] Adding standard routes to API service_providers for `type` (API Prefix: /OS-FEDERATION) [/service_providers, /service_providers/] {{(pid=81577) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 30 01:41:46.498000 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.common [-] Adding standard routes to API mappings for `type` (API Prefix: /OS-FEDERATION) [/mappings, /mappings/] {{(pid=81578) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 30 01:41:46.499989 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//groups//roles//inherited_to_projects' {}] {{(pid=81577) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 01:41:46.500293 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//groups//roles/inherited_to_projects' {}] {{(pid=81577) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 01:41:46.500585 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//users//roles//inherited_to_projects' {}] {{(pid=81577) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 01:41:46.500875 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//users//roles/inherited_to_projects' {}] {{(pid=81577) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 01:41:46.501157 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.common [-] Adding standard routes to API service_providers for `type` (API Prefix: /OS-FEDERATION) [/service_providers, /service_providers/] {{(pid=81578) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 30 01:41:46.501249 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/projects//users//roles//inherited_to_projects' {}] {{(pid=81577) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 01:41:46.501575 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/projects//groups//roles//inherited_to_projects' {}] {{(pid=81577) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 01:41:46.504496 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//groups//roles//inherited_to_projects' {}] {{(pid=81578) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 01:41:46.504831 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//groups//roles/inherited_to_projects' {}] {{(pid=81578) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 01:41:46.505151 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//users//roles//inherited_to_projects' {}] {{(pid=81578) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 01:41:46.505441 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//users//roles/inherited_to_projects' {}] {{(pid=81578) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 01:41:46.505761 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/projects//users//roles//inherited_to_projects' {}] {{(pid=81578) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 01:41:46.506045 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/projects//groups//roles//inherited_to_projects' {}] {{(pid=81578) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 01:41:46.517994 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.common [-] Adding standard routes to API OS-OAUTH1 for `type` (API Prefix: /OS-OAUTH1) [/consumers, /consumers/] {{(pid=81577) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 30 01:41:46.523325 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/request_token' {}] {{(pid=81577) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 01:41:46.523325 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/access_token' {}] {{(pid=81577) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 01:41:46.523325 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/authorize/' {}] {{(pid=81577) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 01:41:46.523600 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.common [-] Adding standard routes to API OS-OAUTH1 for `type` (API Prefix: /OS-OAUTH1) [/consumers, /consumers/] {{(pid=81578) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 30 01:41:46.523600 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/request_token' {}] {{(pid=81578) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 01:41:46.523600 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/access_token' {}] {{(pid=81578) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 01:41:46.523600 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/authorize/' {}] {{(pid=81578) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 01:41:46.525525 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH2: ['/token' {}] {{(pid=81577) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 01:41:46.527743 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.common [-] Adding resource routes to API events: ['/events' {}] {{(pid=81577) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 01:41:46.529268 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-SIMPLE-CERT: ['/OS-SIMPLE-CERT/ca' {}] {{(pid=81577) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 01:41:46.529572 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-SIMPLE-CERT: ['/OS-SIMPLE-CERT/certificates' {}] {{(pid=81577) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 01:41:46.531179 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH2: ['/token' {}] {{(pid=81578) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 01:41:46.532207 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.common [-] Adding standard routes to API policy for `type` (API Prefix: ) [/policies, /policies/] {{(pid=81577) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 30 01:41:46.532494 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/endpoints' {}] {{(pid=81577) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 01:41:46.532709 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.common [-] Adding resource routes to API events: ['/events' {}] {{(pid=81578) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 01:41:46.532791 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/endpoints/' {}] {{(pid=81577) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 01:41:46.533057 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/services/' {}] {{(pid=81577) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 01:41:46.533346 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/services//regions/' {}] {{(pid=81577) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 01:41:46.534359 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-SIMPLE-CERT: ['/OS-SIMPLE-CERT/ca' {}] {{(pid=81578) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 01:41:46.534643 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-SIMPLE-CERT: ['/OS-SIMPLE-CERT/certificates' {}] {{(pid=81578) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 01:41:46.537331 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.common [-] Adding standard routes to API policy for `type` (API Prefix: ) [/policies, /policies/] {{(pid=81578) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 30 01:41:46.537646 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/endpoints' {}] {{(pid=81578) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 01:41:46.538052 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/endpoints/' {}] {{(pid=81578) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 01:41:46.538347 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/services/' {}] {{(pid=81578) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 01:41:46.538638 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/services//regions/' {}] {{(pid=81578) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 01:41:46.545431 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.common [-] Adding standard routes to API projects for `type` (API Prefix: ) [/projects, /projects/] {{(pid=81577) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 30 01:41:46.545732 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//tags' {}] {{(pid=81577) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 01:41:46.546072 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//tags/' {}] {{(pid=81577) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 01:41:46.546298 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//users//roles/' {}] {{(pid=81577) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 01:41:46.546581 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//users//roles' {}] {{(pid=81577) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 01:41:46.546869 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//groups//roles/' {}] {{(pid=81577) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 01:41:46.547162 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//groups//roles' {}] {{(pid=81577) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 01:41:46.551062 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.common [-] Adding standard routes to API projects for `type` (API Prefix: ) [/projects, /projects/] {{(pid=81578) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 30 01:41:46.551968 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//tags' {}] {{(pid=81578) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 01:41:46.551968 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//tags/' {}] {{(pid=81578) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 01:41:46.552170 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//users//roles/' {}] {{(pid=81578) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 01:41:46.552318 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//users//roles' {}] {{(pid=81578) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 01:41:46.552603 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//groups//roles/' {}] {{(pid=81578) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 01:41:46.552948 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//groups//roles' {}] {{(pid=81578) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 01:41:46.563296 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.common [-] Adding standard routes to API regions for `type` (API Prefix: ) [/regions, /regions/] {{(pid=81577) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 30 01:41:46.566343 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.common [-] Adding standard routes to API registered_limit for `type` (API Prefix: ) [/registered_limits, /registered_limits/] {{(pid=81577) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 30 01:41:46.569202 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.common [-] Adding standard routes to API regions for `type` (API Prefix: ) [/regions, /regions/] {{(pid=81578) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 30 01:41:46.569288 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.common [-] Adding resource routes to API role_assignments: ['/role_assignments' {}] {{(pid=81577) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 01:41:46.570678 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.common [-] Adding resource routes to API role_inferences: ['/role_inferences' {}] {{(pid=81577) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 01:41:46.572158 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.common [-] Adding standard routes to API roles for `type` (API Prefix: ) [/roles, /roles/] {{(pid=81577) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 30 01:41:46.572240 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.common [-] Adding standard routes to API registered_limit for `type` (API Prefix: ) [/registered_limits, /registered_limits/] {{(pid=81578) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 30 01:41:46.572441 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.common [-] Adding resource routes to API roles: ['/roles//implies' {}] {{(pid=81577) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 01:41:46.572742 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.common [-] Adding resource routes to API roles: ['/roles//implies/' {}] {{(pid=81577) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 01:41:46.575321 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.common [-] Adding resource routes to API role_assignments: ['/role_assignments' {}] {{(pid=81578) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 01:41:46.576774 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.common [-] Adding resource routes to API role_inferences: ['/role_inferences' {}] {{(pid=81578) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 01:41:46.578409 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.common [-] Adding standard routes to API roles for `type` (API Prefix: ) [/roles, /roles/] {{(pid=81578) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 30 01:41:46.578699 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.common [-] Adding resource routes to API roles: ['/roles//implies' {}] {{(pid=81578) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 01:41:46.579004 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.common [-] Adding resource routes to API roles: ['/roles//implies/' {}] {{(pid=81578) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 01:41:46.579600 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.common [-] Adding resource routes to API s3tokens: ['/s3tokens' {}] {{(pid=81577) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 01:41:46.581435 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.common [-] Adding standard routes to API services for `type` (API Prefix: ) [/services, /services/] {{(pid=81577) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 30 01:41:46.584216 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/users//roles' {}] {{(pid=81577) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 01:41:46.584521 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/users//roles/' {}] {{(pid=81577) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 01:41:46.584811 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/groups//roles' {}] {{(pid=81577) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 01:41:46.585099 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/groups//roles/' {}] {{(pid=81577) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 01:41:46.585541 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.common [-] Adding resource routes to API s3tokens: ['/s3tokens' {}] {{(pid=81578) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 01:41:46.586956 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.common [-] Adding standard routes to API services for `type` (API Prefix: ) [/services, /services/] {{(pid=81578) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 30 01:41:46.590343 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/users//roles' {}] {{(pid=81578) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 01:41:46.590663 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/users//roles/' {}] {{(pid=81578) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 01:41:46.590952 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/groups//roles' {}] {{(pid=81578) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 01:41:46.591234 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/groups//roles/' {}] {{(pid=81578) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 01:41:46.593068 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.common [-] Adding standard routes to API trusts for `type` (API Prefix: /OS-TRUST) [/trusts, /trusts/] {{(pid=81577) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 30 01:41:46.593358 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.common [-] Adding resource routes to API trusts: ['/trusts//roles' {}] {{(pid=81577) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 01:41:46.593673 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.common [-] Adding resource routes to API trusts: ['/trusts//roles/' {}] {{(pid=81577) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 01:41:46.599206 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.common [-] Adding standard routes to API trusts for `type` (API Prefix: /OS-TRUST) [/trusts, /trusts/] {{(pid=81578) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 30 01:41:46.599501 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.common [-] Adding resource routes to API trusts: ['/trusts//roles' {}] {{(pid=81578) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 01:41:46.599789 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.common [-] Adding resource routes to API trusts: ['/trusts//roles/' {}] {{(pid=81578) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 01:41:46.600548 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.common [-] Adding standard routes to API users for `type` (API Prefix: ) [/users, /users/] {{(pid=81577) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 30 01:41:46.601097 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//password' {}] {{(pid=81577) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 01:41:46.601380 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//groups' {}] {{(pid=81577) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 01:41:46.601680 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//projects' {}] {{(pid=81577) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 01:41:46.601958 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//credentials/OS-EC2' {}] {{(pid=81577) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 01:41:46.602231 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//credentials/OS-EC2/' {}] {{(pid=81577) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 01:41:46.602515 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens' {}] {{(pid=81577) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 01:41:46.602791 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens/' {}] {{(pid=81577) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 01:41:46.603070 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens//roles' {}] {{(pid=81577) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 01:41:46.603346 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens//roles/' {}] {{(pid=81577) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 01:41:46.603617 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//application_credentials' {}] {{(pid=81577) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 01:41:46.603878 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//application_credentials/' {}] {{(pid=81577) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 01:41:46.604671 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//access_rules' {}] {{(pid=81577) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 01:41:46.604954 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//access_rules/' {}] {{(pid=81577) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 01:41:46.607735 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.common [-] Adding standard routes to API users for `type` (API Prefix: ) [/users, /users/] {{(pid=81578) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 30 01:41:46.608018 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//password' {}] {{(pid=81578) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 01:41:46.608301 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//groups' {}] {{(pid=81578) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 01:41:46.608586 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//projects' {}] {{(pid=81578) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 01:41:46.608862 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//credentials/OS-EC2' {}] {{(pid=81578) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 01:41:46.609289 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//credentials/OS-EC2/' {}] {{(pid=81578) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 01:41:46.609450 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens' {}] {{(pid=81578) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 01:41:46.609757 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens/' {}] {{(pid=81578) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 01:41:46.610042 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens//roles' {}] {{(pid=81578) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 01:41:46.610322 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens//roles/' {}] {{(pid=81578) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 01:41:46.610597 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//application_credentials' {}] {{(pid=81578) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 01:41:46.610860 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//application_credentials/' {}] {{(pid=81578) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 01:41:46.611128 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//access_rules' {}] {{(pid=81578) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 01:41:46.611412 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//access_rules/' {}] {{(pid=81578) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 30 01:41:46.639594 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: WSGI app 0 (mountpoint='') ready in 5 seconds on interpreter 0x562424ee09c0 pid: 81577 (default app) Aug 30 01:41:46.643100 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73f9cb69-ffb6-4709-b010-fd38e11a8c6b None None] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:41:46.643393 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73f9cb69-ffb6-4709-b010-fd38e11a8c6b None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:41:46.643666 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73f9cb69-ffb6-4709-b010-fd38e11a8c6b None None] PATH_INFO: `/v3` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:41:46.647094 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: WSGI app 0 (mountpoint='') ready in 5 seconds on interpreter 0x562424ee09c0 pid: 81578 (default app) Aug 30 01:41:46.647863 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1/1] 10.222.0.27 () {50 vars in 791 bytes} [Tue Aug 30 01:41:46 2022] GET /identity/v3/ => generated 255 bytes in 8 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:41:54.220656 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b284eb6-3234-4aea-bd46-1c43a78e7316 None None] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:41:54.221179 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b284eb6-3234-4aea-bd46-1c43a78e7316 None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:41:54.221341 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b284eb6-3234-4aea-bd46-1c43a78e7316 None None] PATH_INFO: `/` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:41:54.226391 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1/2] 10.222.0.27 () {54 vars in 927 bytes} [Tue Aug 30 01:41:54 2022] GET /identity => generated 270 bytes in 10 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 30 01:41:54.232357 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0680ba89-bd4f-4c84-a78d-51b38ceca1aa None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:41:54.232829 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0680ba89-bd4f-4c84-a78d-51b38ceca1aa None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:41:54.232928 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0680ba89-bd4f-4c84-a78d-51b38ceca1aa None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:41:54.302740 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: INFO dbcounter [None req-0680ba89-bd4f-4c84-a78d-51b38ceca1aa None None] Registered counter for database keystone Aug 30 01:41:54.322138 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG oslo_db.sqlalchemy.engines [None req-0680ba89-bd4f-4c84-a78d-51b38ceca1aa None None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=81577) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Aug 30 01:41:54.329028 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writer thread running {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:99}} Aug 30 01:41:54.511134 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG passlib.handlers.bcrypt [None req-0680ba89-bd4f-4c84-a78d-51b38ceca1aa None None] detected 'bcrypt' backend, version '4.0.0' {{(pid=81577) _load_backend_mixin /usr/local/lib/python3.8/dist-packages/passlib/handlers/bcrypt.py:625}} Aug 30 01:41:54.511590 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG passlib.handlers.bcrypt [None req-0680ba89-bd4f-4c84-a78d-51b38ceca1aa None None] 'bcrypt' backend lacks $2$ support, enabling workaround {{(pid=81577) _finalize_backend_mixin /usr/local/lib/python3.8/dist-packages/passlib/handlers/bcrypt.py:406}} Aug 30 01:41:54.569420 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-0680ba89-bd4f-4c84-a78d-51b38ceca1aa None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 01:41:54.571054 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-0680ba89-bd4f-4c84-a78d-51b38ceca1aa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:41:54.703178 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 2/3] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Aug 30 01:41:54 2022] POST /identity/v3/auth/tokens => generated 914 bytes in 471 msecs (HTTP/1.1 201) 6 headers in 384 bytes (1 switches on core 0) Aug 30 01:41:55.155763 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b284eb6-3234-4aea-bd46-1c43a78e7316 None None] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:41:55.226449 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: INFO dbcounter [None req-6b284eb6-3234-4aea-bd46-1c43a78e7316 None None] Registered counter for database keystone Aug 30 01:41:55.248248 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG oslo_db.sqlalchemy.engines [None req-6b284eb6-3234-4aea-bd46-1c43a78e7316 None None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=81578) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Aug 30 01:41:55.254474 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writer thread running {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:99}} Aug 30 01:41:55.373828 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b284eb6-3234-4aea-bd46-1c43a78e7316 None None] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:41:55.378011 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e148ca1c-893b-4689-8284-ac0adb230498 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:41:55.378632 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e148ca1c-893b-4689-8284-ac0adb230498 admin admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:41:55.378907 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e148ca1c-893b-4689-8284-ac0adb230498 admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:41:55.379172 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e148ca1c-893b-4689-8284-ac0adb230498 admin admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:41:55.387306 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e148ca1c-893b-4689-8284-ac0adb230498 admin admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:41:55.438820 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: WARNING keystone.common.rbac_enforcer.enforcer [None req-e148ca1c-893b-4689-8284-ac0adb230498 admin admin] Deprecated policy rules found. Use oslopolicy-policy-generator and oslopolicy-policy-upgrade to detect and resolve deprecated policies in your configuration. Aug 30 01:41:55.439092 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e148ca1c-893b-4689-8284-ac0adb230498 admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:41:55.448613 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 2/4] 10.222.0.27 () {58 vars in 1324 bytes} [Tue Aug 30 01:41:55 2022] GET /identity/v3/auth/tokens => generated 914 bytes in 293 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Aug 30 01:41:55.454970 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0680ba89-bd4f-4c84-a78d-51b38ceca1aa None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:41:55.469299 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0680ba89-bd4f-4c84-a78d-51b38ceca1aa None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:41:55.473808 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec3bdcd1-cb5b-4ae4-9c24-a80b880f76f7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:41:55.474917 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec3bdcd1-cb5b-4ae4-9c24-a80b880f76f7 admin admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:41:55.475196 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec3bdcd1-cb5b-4ae4-9c24-a80b880f76f7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:41:55.475507 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec3bdcd1-cb5b-4ae4-9c24-a80b880f76f7 admin admin] PATH_INFO: `/v3/projects/70bc9f42895844308fb4b9192314f80b` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:41:55.477360 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec3bdcd1-cb5b-4ae4-9c24-a80b880f76f7 admin admin] RBAC: Authorizing `identity:get_project(project_id=70bc9f42895844308fb4b9192314f80b)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:41:55.530759 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: WARNING keystone.common.rbac_enforcer.enforcer [None req-ec3bdcd1-cb5b-4ae4-9c24-a80b880f76f7 admin admin] Deprecated policy rules found. Use oslopolicy-policy-generator and oslopolicy-policy-upgrade to detect and resolve deprecated policies in your configuration. Aug 30 01:41:55.531094 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec3bdcd1-cb5b-4ae4-9c24-a80b880f76f7 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:41:55.535888 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 3/5] 10.222.0.27 () {56 vars in 1207 bytes} [Tue Aug 30 01:41:55 2022] GET /identity/v3/projects/70bc9f42895844308fb4b9192314f80b => generated 344 bytes in 82 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:41:57.421170 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2927d66-e3e0-416c-8635-9c9532ef60fe None None] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:41:57.421758 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2927d66-e3e0-416c-8635-9c9532ef60fe None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:41:57.421758 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2927d66-e3e0-416c-8635-9c9532ef60fe None None] PATH_INFO: `/` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:41:57.425844 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 3/6] 10.222.0.27 () {54 vars in 927 bytes} [Tue Aug 30 01:41:57 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 01:41:57.431325 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4259223d-7e71-4281-ab8e-e731f14d260c None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:41:57.431713 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4259223d-7e71-4281-ab8e-e731f14d260c None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:41:57.431894 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4259223d-7e71-4281-ab8e-e731f14d260c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:41:57.453206 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-4259223d-7e71-4281-ab8e-e731f14d260c None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 01:41:57.454756 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-4259223d-7e71-4281-ab8e-e731f14d260c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:41:57.462578 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 4/7] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Aug 30 01:41:57 2022] POST /identity/v3/auth/tokens => generated 914 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 384 bytes (1 switches on core 0) Aug 30 01:41:57.734048 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2927d66-e3e0-416c-8635-9c9532ef60fe None None] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:41:57.748223 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2927d66-e3e0-416c-8635-9c9532ef60fe None None] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:41:57.752381 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6a80b91-08ad-44ce-b854-41d4751442a7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:41:57.752972 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6a80b91-08ad-44ce-b854-41d4751442a7 admin admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:41:57.753248 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6a80b91-08ad-44ce-b854-41d4751442a7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:41:57.753545 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6a80b91-08ad-44ce-b854-41d4751442a7 admin admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:41:57.761436 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6a80b91-08ad-44ce-b854-41d4751442a7 admin admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:41:57.763201 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6a80b91-08ad-44ce-b854-41d4751442a7 admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:41:57.771676 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 4/8] 10.222.0.27 () {58 vars in 1324 bytes} [Tue Aug 30 01:41:57 2022] GET /identity/v3/auth/tokens => generated 914 bytes in 38 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Aug 30 01:41:57.777180 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4259223d-7e71-4281-ab8e-e731f14d260c None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:41:57.786235 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4259223d-7e71-4281-ab8e-e731f14d260c None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:41:57.790581 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-946f1bbb-7803-460b-85c6-0e266ce6149f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:41:57.791757 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-946f1bbb-7803-460b-85c6-0e266ce6149f admin admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:41:57.792017 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-946f1bbb-7803-460b-85c6-0e266ce6149f admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:41:57.792289 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-946f1bbb-7803-460b-85c6-0e266ce6149f admin admin] PATH_INFO: `/v3/users/ac83d5ed111c4f4e8b3c3226bd3bbaba` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:41:57.793645 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-946f1bbb-7803-460b-85c6-0e266ce6149f admin admin] RBAC: Authorizing `identity:get_user(user_id=ac83d5ed111c4f4e8b3c3226bd3bbaba)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:41:57.795470 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-946f1bbb-7803-460b-85c6-0e266ce6149f admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:41:57.797018 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 5/9] 10.222.0.27 () {56 vars in 1198 bytes} [Tue Aug 30 01:41:57 2022] GET /identity/v3/users/ac83d5ed111c4f4e8b3c3226bd3bbaba => generated 247 bytes in 20 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 30 01:41:59.780870 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ff30861-64d0-4141-86c5-20fdb5a61faf None None] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:41:59.781408 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ff30861-64d0-4141-86c5-20fdb5a61faf None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:41:59.781408 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ff30861-64d0-4141-86c5-20fdb5a61faf None None] PATH_INFO: `/` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:41:59.782746 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 5/10] 10.222.0.27 () {54 vars in 927 bytes} [Tue Aug 30 01:41:59 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 01:41:59.787963 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce9c0aa8-1336-4acc-a65f-663563f11478 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:41:59.788361 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce9c0aa8-1336-4acc-a65f-663563f11478 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:41:59.788518 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce9c0aa8-1336-4acc-a65f-663563f11478 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:41:59.806828 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-ce9c0aa8-1336-4acc-a65f-663563f11478 None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 01:41:59.808045 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-ce9c0aa8-1336-4acc-a65f-663563f11478 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:41:59.821762 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5113ebb-8a4a-49d6-9b22-cedeb693af8e None None] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:41:59.822053 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5113ebb-8a4a-49d6-9b22-cedeb693af8e None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:41:59.822331 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5113ebb-8a4a-49d6-9b22-cedeb693af8e None None] PATH_INFO: `/` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:41:59.824481 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 6/11] 10.222.0.27 () {54 vars in 927 bytes} [Tue Aug 30 01:41:59 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 30 01:41:59.829854 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6b343cc-e598-43b0-966b-fa3918b5cea9 None None] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:41:59.830139 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6b343cc-e598-43b0-966b-fa3918b5cea9 None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:41:59.830413 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6b343cc-e598-43b0-966b-fa3918b5cea9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:41:59.873641 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG passlib.handlers.bcrypt [None req-d6b343cc-e598-43b0-966b-fa3918b5cea9 None None] detected 'bcrypt' backend, version '4.0.0' {{(pid=81578) _load_backend_mixin /usr/local/lib/python3.8/dist-packages/passlib/handlers/bcrypt.py:625}} Aug 30 01:41:59.874027 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG passlib.handlers.bcrypt [None req-d6b343cc-e598-43b0-966b-fa3918b5cea9 None None] 'bcrypt' backend lacks $2$ support, enabling workaround {{(pid=81578) _finalize_backend_mixin /usr/local/lib/python3.8/dist-packages/passlib/handlers/bcrypt.py:406}} Aug 30 01:41:59.888088 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 6/12] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Aug 30 01:41:59 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 30 01:41:59.922944 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.auth.core [None req-d6b343cc-e598-43b0-966b-fa3918b5cea9 None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81578) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 01:41:59.924556 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-d6b343cc-e598-43b0-966b-fa3918b5cea9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:42:00.030363 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 7/13] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Aug 30 01:41:59 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 201 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Aug 30 01:42:00.154683 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce9c0aa8-1336-4acc-a65f-663563f11478 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:42:00.179921 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:42:00.264532 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce9c0aa8-1336-4acc-a65f-663563f11478 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:42:00.298276 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9f0636c-8026-4b67-beac-542f6a8d101a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:42:00.299228 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9f0636c-8026-4b67-beac-542f6a8d101a None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:00.299508 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9f0636c-8026-4b67-beac-542f6a8d101a None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:00.299779 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9f0636c-8026-4b67-beac-542f6a8d101a None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:00.300082 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6b343cc-e598-43b0-966b-fa3918b5cea9 None None] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:42:00.311397 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9f0636c-8026-4b67-beac-542f6a8d101a None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 30 01:42:00.320144 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application [None req-d9f0636c-8026-4b67-beac-542f6a8d101a None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 30 01:42:00.320144 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 01:42:00.320144 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 01:42:00.320144 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 01:42:00.320144 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 01:42:00.320144 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 01:42:00.320144 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 01:42:00.320144 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 01:42:00.320144 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 01:42:00.320144 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 01:42:00.320144 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 01:42:00.320144 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 01:42:00.320144 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 30 01:42:00.320144 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 30 01:42:00.320144 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 30 01:42:00.320144 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 30 01:42:00.321077 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 30 01:42:00.321077 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 30 01:42:00.321077 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 30 01:42:00.321077 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 30 01:42:00.321077 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 01:42:00.321077 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 01:42:00.321077 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 01:42:00.321077 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 01:42:00.321077 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 01:42:00.321077 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 01:42:00.321077 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 01:42:00.321077 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application with Lock( Aug 30 01:42:00.321077 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 01:42:00.321077 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._enter() Aug 30 01:42:00.321077 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 01:42:00.321077 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 01:42:00.321916 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 01:42:00.321916 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.creator() Aug 30 01:42:00.321916 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 01:42:00.321916 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application created_value = creator( Aug 30 01:42:00.321916 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 30 01:42:00.321916 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 30 01:42:00.321916 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 30 01:42:00.321916 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 30 01:42:00.321916 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 30 01:42:00.321916 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 30 01:42:00.321916 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 30 01:42:00.321916 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Aug 30 01:42:00.321916 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 7/14] 10.222.0.27 () {56 vars in 1096 bytes} [Tue Aug 30 01:42:00 2022] GET /identity/v3/roles/admin => generated 83 bytes in 167 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 01:42:00.327589 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9f0636c-8026-4b67-beac-542f6a8d101a None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:42:00.403212 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6b343cc-e598-43b0-966b-fa3918b5cea9 None None] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:42:00.405845 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:42:00.410719 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:42:00.428969 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9f0636c-8026-4b67-beac-542f6a8d101a None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:42:00.457709 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e25d7edb-e68a-4662-978e-e62d896f97a9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:42:00.457709 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e25d7edb-e68a-4662-978e-e62d896f97a9 None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:00.457709 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e25d7edb-e68a-4662-978e-e62d896f97a9 None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:00.457709 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e25d7edb-e68a-4662-978e-e62d896f97a9 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:00.462536 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c463cebb-665e-44f4-b740-935c5bdbfa19 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:42:00.463677 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c463cebb-665e-44f4-b740-935c5bdbfa19 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:00.464019 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c463cebb-665e-44f4-b740-935c5bdbfa19 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:00.464292 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c463cebb-665e-44f4-b740-935c5bdbfa19 None admin] PATH_INFO: `/v3/roles` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:00.465009 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c463cebb-665e-44f4-b740-935c5bdbfa19 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:42:00.465311 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c463cebb-665e-44f4-b740-935c5bdbfa19 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:42:00.466743 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c463cebb-665e-44f4-b740-935c5bdbfa19 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:42:00.478736 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 8/15] 10.222.0.27 () {56 vars in 1099 bytes} [Tue Aug 30 01:42:00 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 30 01:42:00.484855 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c463cebb-665e-44f4-b740-935c5bdbfa19 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:42:00.510179 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e25d7edb-e68a-4662-978e-e62d896f97a9 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:42:00.511700 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e25d7edb-e68a-4662-978e-e62d896f97a9 None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:42:00.590210 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c463cebb-665e-44f4-b740-935c5bdbfa19 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:42:00.616997 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 8/16] 10.222.0.27 () {58 vars in 1282 bytes} [Tue Aug 30 01:42:00 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 317 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Aug 30 01:42:00.623371 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e25d7edb-e68a-4662-978e-e62d896f97a9 None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:42:00.626527 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:42:00.630813 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29b3c6d5-9e1d-4298-bd43-df9ff3a4f58d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:42:00.631452 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29b3c6d5-9e1d-4298-bd43-df9ff3a4f58d None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:00.631766 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29b3c6d5-9e1d-4298-bd43-df9ff3a4f58d None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:00.632055 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29b3c6d5-9e1d-4298-bd43-df9ff3a4f58d None admin] PATH_INFO: `/v3/users/ac83d5ed111c4f4e8b3c3226bd3bbaba` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:00.633222 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29b3c6d5-9e1d-4298-bd43-df9ff3a4f58d None admin] RBAC: Authorizing `identity:get_user(user_id=ac83d5ed111c4f4e8b3c3226bd3bbaba)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:42:00.634817 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29b3c6d5-9e1d-4298-bd43-df9ff3a4f58d None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:42:00.649641 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 9/17] 10.222.0.27 () {56 vars in 1177 bytes} [Tue Aug 30 01:42:00 2022] GET /identity/v3/users/ac83d5ed111c4f4e8b3c3226bd3bbaba => generated 247 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 30 01:42:00.649641 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29b3c6d5-9e1d-4298-bd43-df9ff3a4f58d None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:42:00.658999 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:42:00.725443 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e25d7edb-e68a-4662-978e-e62d896f97a9 None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:42:00.754529 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29b3c6d5-9e1d-4298-bd43-df9ff3a4f58d None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:42:00.761849 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4dae9738-18b5-4012-a5e8-7e7eac9c453b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:42:00.762705 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dae9738-18b5-4012-a5e8-7e7eac9c453b None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:00.762985 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dae9738-18b5-4012-a5e8-7e7eac9c453b None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:00.763254 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dae9738-18b5-4012-a5e8-7e7eac9c453b None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:00.782304 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4dae9738-18b5-4012-a5e8-7e7eac9c453b None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:42:00.783792 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4dae9738-18b5-4012-a5e8-7e7eac9c453b None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:42:00.789167 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88545f7b-d2f8-4219-adef-8a030c0c1f35 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:42:00.789830 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88545f7b-d2f8-4219-adef-8a030c0c1f35 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:00.790109 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88545f7b-d2f8-4219-adef-8a030c0c1f35 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:00.790381 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88545f7b-d2f8-4219-adef-8a030c0c1f35 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:00.792009 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88545f7b-d2f8-4219-adef-8a030c0c1f35 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:42:00.793639 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88545f7b-d2f8-4219-adef-8a030c0c1f35 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:42:00.795181 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 10/18] 10.222.0.27 () {56 vars in 1108 bytes} [Tue Aug 30 01:42:00 2022] GET /identity/v3/domains/default => generated 199 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 30 01:42:00.801005 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88545f7b-d2f8-4219-adef-8a030c0c1f35 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:42:00.802753 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application [None req-4dae9738-18b5-4012-a5e8-7e7eac9c453b None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 01:42:00.802753 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 01:42:00.802753 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 01:42:00.802753 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 01:42:00.802753 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 01:42:00.802753 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 01:42:00.802753 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 01:42:00.802753 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 01:42:00.802753 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 01:42:00.802753 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 01:42:00.802753 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 01:42:00.802753 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 01:42:00.802753 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 30 01:42:00.802753 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 30 01:42:00.802753 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 30 01:42:00.802753 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 30 01:42:00.803760 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 01:42:00.803760 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 01:42:00.803760 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 01:42:00.803760 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 01:42:00.803760 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 01:42:00.803760 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 01:42:00.803760 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 01:42:00.803760 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application with Lock( Aug 30 01:42:00.803760 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 01:42:00.803760 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._enter() Aug 30 01:42:00.803760 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 01:42:00.803760 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 01:42:00.803760 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 01:42:00.803760 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.creator() Aug 30 01:42:00.803760 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 01:42:00.803760 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application created_value = creator( Aug 30 01:42:00.803760 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 30 01:42:00.804602 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 30 01:42:00.804602 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 01:42:00.804602 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application Aug 30 01:42:00.804602 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 9/19] 10.222.0.27 () {56 vars in 1108 bytes} [Tue Aug 30 01:42:00 2022] GET /identity/v3/domains/Default => generated 87 bytes in 181 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 30 01:42:00.810349 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4dae9738-18b5-4012-a5e8-7e7eac9c453b None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:42:00.868064 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:42:00.883838 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:42:00.904009 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88545f7b-d2f8-4219-adef-8a030c0c1f35 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:42:00.912647 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4dae9738-18b5-4012-a5e8-7e7eac9c453b None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:42:00.940153 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61934504-f52e-457a-87dd-41de9dcaafe7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:42:00.940760 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61934504-f52e-457a-87dd-41de9dcaafe7 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:00.941038 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61934504-f52e-457a-87dd-41de9dcaafe7 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:00.941305 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61934504-f52e-457a-87dd-41de9dcaafe7 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:00.942060 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61934504-f52e-457a-87dd-41de9dcaafe7 None admin] RBAC: Adding query filter params (role.id=ead5fac86a0747ea8823a82efcea8ed7, scope.domain.id=default, user.id=ac83d5ed111c4f4e8b3c3226bd3bbaba) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:42:00.942372 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61934504-f52e-457a-87dd-41de9dcaafe7 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:42:00.943796 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61934504-f52e-457a-87dd-41de9dcaafe7 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:42:00.947247 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69e0c36a-da3a-43ad-a807-25481835eea7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:42:00.947865 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69e0c36a-da3a-43ad-a807-25481835eea7 None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:00.948146 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69e0c36a-da3a-43ad-a807-25481835eea7 None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:00.948441 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69e0c36a-da3a-43ad-a807-25481835eea7 None admin] PATH_INFO: `/v3/domains` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:00.949088 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69e0c36a-da3a-43ad-a807-25481835eea7 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81578) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:42:00.949390 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69e0c36a-da3a-43ad-a807-25481835eea7 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:42:00.950525 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 11/20] 10.222.0.27 () {56 vars in 1322 bytes} [Tue Aug 30 01:42:00 2022] GET /identity/v3/role_assignments?user.id=ac83d5ed111c4f4e8b3c3226bd3bbaba&scope.domain.id=default&role.id=ead5fac86a0747ea8823a82efcea8ed7 => generated 233 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 30 01:42:00.950781 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69e0c36a-da3a-43ad-a807-25481835eea7 None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:42:00.970349 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 10/21] 10.222.0.27 () {56 vars in 1109 bytes} [Tue Aug 30 01:42:00 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 30 01:42:03.281682 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2b34efc-d10a-4db0-8d57-1c04edb961e2 None None] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:03.282717 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2b34efc-d10a-4db0-8d57-1c04edb961e2 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:03.283212 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2b34efc-d10a-4db0-8d57-1c04edb961e2 None None] PATH_INFO: `/` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:03.284751 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 12/22] 10.222.0.27 () {54 vars in 927 bytes} [Tue Aug 30 01:42:03 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 01:42:03.290125 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1720470d-a4dc-49d9-87ab-b00c1ba5a434 None None] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:03.291002 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1720470d-a4dc-49d9-87ab-b00c1ba5a434 None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:03.292035 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1720470d-a4dc-49d9-87ab-b00c1ba5a434 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:03.323065 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.auth.core [None req-1720470d-a4dc-49d9-87ab-b00c1ba5a434 None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81578) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 01:42:03.324629 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-1720470d-a4dc-49d9-87ab-b00c1ba5a434 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:42:03.420584 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 11/23] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Aug 30 01:42:03 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 131 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Aug 30 01:42:03.686078 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99122735-9898-4b07-aecc-1ef7494b6193 None None] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:03.686749 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99122735-9898-4b07-aecc-1ef7494b6193 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:03.687230 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99122735-9898-4b07-aecc-1ef7494b6193 None None] PATH_INFO: `/` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:03.688792 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 13/24] 10.222.0.27 () {54 vars in 927 bytes} [Tue Aug 30 01:42:03 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 01:42:03.711082 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0c375b9-9722-40ff-b611-e2cc205eb57e None None] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:03.711725 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0c375b9-9722-40ff-b611-e2cc205eb57e None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:03.712205 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0c375b9-9722-40ff-b611-e2cc205eb57e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:03.743445 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.auth.core [None req-f0c375b9-9722-40ff-b611-e2cc205eb57e None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81578) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 01:42:03.744876 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-f0c375b9-9722-40ff-b611-e2cc205eb57e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:42:03.804348 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:42:03.817709 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88a02f5d-0882-4ba7-b390-050424966373 None None] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:03.818004 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88a02f5d-0882-4ba7-b390-050424966373 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:03.818271 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88a02f5d-0882-4ba7-b390-050424966373 None None] PATH_INFO: `/` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:03.819615 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 14/25] 10.222.0.27 () {54 vars in 927 bytes} [Tue Aug 30 01:42:03 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 30 01:42:03.826736 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4744c347-9cb7-42b0-b7e0-1f51eb27853a None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:03.827922 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4744c347-9cb7-42b0-b7e0-1f51eb27853a None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:03.827922 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4744c347-9cb7-42b0-b7e0-1f51eb27853a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:03.851947 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 12/26] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Aug 30 01:42:03 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 142 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 30 01:42:03.857977 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-4744c347-9cb7-42b0-b7e0-1f51eb27853a None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 01:42:03.859279 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-4744c347-9cb7-42b0-b7e0-1f51eb27853a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:42:03.892675 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0c375b9-9722-40ff-b611-e2cc205eb57e None None] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:42:03.965267 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 15/27] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Aug 30 01:42:03 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 139 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 30 01:42:04.023802 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f266ce62-9ade-49b3-9694-8576bee63da1 None None] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:04.024363 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f266ce62-9ade-49b3-9694-8576bee63da1 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:04.024840 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f266ce62-9ade-49b3-9694-8576bee63da1 None None] PATH_INFO: `/` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:04.027411 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 16/28] 10.222.0.27 () {54 vars in 927 bytes} [Tue Aug 30 01:42:04 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 01:42:04.032098 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09508d3b-1114-47db-b130-99c480aeb24c None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:04.032625 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09508d3b-1114-47db-b130-99c480aeb24c None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:04.033110 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09508d3b-1114-47db-b130-99c480aeb24c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:04.061005 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-09508d3b-1114-47db-b130-99c480aeb24c None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 01:42:04.066656 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-09508d3b-1114-47db-b130-99c480aeb24c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:42:04.100229 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0c375b9-9722-40ff-b611-e2cc205eb57e None None] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:42:04.179079 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3cae694-5df4-424c-8ba9-66746d6f1931 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:42:04.179760 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3cae694-5df4-424c-8ba9-66746d6f1931 None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:04.180021 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3cae694-5df4-424c-8ba9-66746d6f1931 None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:04.180294 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3cae694-5df4-424c-8ba9-66746d6f1931 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:04.189984 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 17/29] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Aug 30 01:42:04 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 158 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 30 01:42:04.202656 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3cae694-5df4-424c-8ba9-66746d6f1931 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:42:04.204152 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3cae694-5df4-424c-8ba9-66746d6f1931 None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:42:04.220534 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application [None req-b3cae694-5df4-424c-8ba9-66746d6f1931 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 01:42:04.220534 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 01:42:04.220534 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 01:42:04.220534 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 01:42:04.220534 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 01:42:04.220534 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 01:42:04.220534 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 01:42:04.220534 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 01:42:04.220534 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 01:42:04.220534 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 01:42:04.220534 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 01:42:04.220534 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 01:42:04.220534 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 30 01:42:04.220534 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 30 01:42:04.220534 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 30 01:42:04.220534 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 30 01:42:04.221419 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 01:42:04.221419 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 01:42:04.221419 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 01:42:04.221419 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 01:42:04.221419 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 01:42:04.221419 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 01:42:04.221419 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 01:42:04.221419 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application with Lock( Aug 30 01:42:04.221419 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 01:42:04.221419 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._enter() Aug 30 01:42:04.221419 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 01:42:04.221419 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 01:42:04.221419 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 01:42:04.221419 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.creator() Aug 30 01:42:04.221419 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 01:42:04.221419 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application created_value = creator( Aug 30 01:42:04.221419 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 30 01:42:04.222344 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 30 01:42:04.222344 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 01:42:04.222344 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application Aug 30 01:42:04.229994 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 13/30] 10.222.0.27 () {56 vars in 1108 bytes} [Tue Aug 30 01:42:03 2022] GET /identity/v3/domains/Default => generated 87 bytes in 338 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 30 01:42:04.238009 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09508d3b-1114-47db-b130-99c480aeb24c None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:42:04.245527 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:42:04.260745 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3cae694-5df4-424c-8ba9-66746d6f1931 None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:42:04.295320 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:42:04.411336 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3cae694-5df4-424c-8ba9-66746d6f1931 None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:42:04.455497 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09508d3b-1114-47db-b130-99c480aeb24c None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:42:04.461367 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e49163d-e3e9-4931-9bce-b64904eccd94 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:42:04.462316 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e49163d-e3e9-4931-9bce-b64904eccd94 None admin] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:04.462825 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e49163d-e3e9-4931-9bce-b64904eccd94 None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:04.463856 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e49163d-e3e9-4931-9bce-b64904eccd94 None admin] PATH_INFO: `/v3/roles` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:04.464748 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e49163d-e3e9-4931-9bce-b64904eccd94 None admin] RBAC: Authorizing `identity:create_role()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:42:04.466407 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e49163d-e3e9-4931-9bce-b64904eccd94 None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:42:04.504085 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c609143-fa47-4cab-bfe0-051f32833018 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:42:04.505060 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c609143-fa47-4cab-bfe0-051f32833018 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:04.505621 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c609143-fa47-4cab-bfe0-051f32833018 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:04.506107 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c609143-fa47-4cab-bfe0-051f32833018 None admin] PATH_INFO: `/v3/domains` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:04.506962 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c609143-fa47-4cab-bfe0-051f32833018 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:42:04.507498 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c609143-fa47-4cab-bfe0-051f32833018 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:42:04.509116 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c609143-fa47-4cab-bfe0-051f32833018 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:42:04.528573 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 18/31] 10.222.0.27 () {56 vars in 1109 bytes} [Tue Aug 30 01:42:04 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 291 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 30 01:42:04.529756 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c609143-fa47-4cab-bfe0-051f32833018 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:42:04.614429 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:42:04.654148 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c609143-fa47-4cab-bfe0-051f32833018 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:42:04.685493 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 14/32] 10.222.0.27 () {60 vars in 1131 bytes} [Tue Aug 30 01:42:04 2022] POST /identity/v3/roles => generated 225 bytes in 426 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 30 01:42:04.686955 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50fb9c86-9bb8-469a-8773-fb79c6336142 None None] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:04.687252 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50fb9c86-9bb8-469a-8773-fb79c6336142 None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:04.687569 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50fb9c86-9bb8-469a-8773-fb79c6336142 None None] PATH_INFO: `/` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:04.689030 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 15/33] 10.222.0.27 () {54 vars in 927 bytes} [Tue Aug 30 01:42:04 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 01:42:04.693612 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50fb9c86-9bb8-469a-8773-fb79c6336142 None None] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:42:04.693822 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c175f303-09c9-4158-9cd4-bfd037012d24 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:42:04.694359 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c175f303-09c9-4158-9cd4-bfd037012d24 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:04.694359 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c175f303-09c9-4158-9cd4-bfd037012d24 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:04.694655 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c175f303-09c9-4158-9cd4-bfd037012d24 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:04.714248 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c175f303-09c9-4158-9cd4-bfd037012d24 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 30 01:42:04.721199 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application [None req-c175f303-09c9-4158-9cd4-bfd037012d24 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 30 01:42:04.721199 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 01:42:04.721199 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 01:42:04.721199 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 01:42:04.721199 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 01:42:04.721199 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 01:42:04.721199 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 01:42:04.721199 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 01:42:04.721199 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 01:42:04.721199 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 01:42:04.721199 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 01:42:04.721199 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 01:42:04.721199 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 30 01:42:04.721199 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 30 01:42:04.721199 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 30 01:42:04.721199 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 30 01:42:04.722087 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 30 01:42:04.722087 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 30 01:42:04.722087 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 30 01:42:04.722087 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 30 01:42:04.722087 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 01:42:04.722087 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 01:42:04.722087 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 01:42:04.722087 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 01:42:04.722087 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 01:42:04.722087 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 01:42:04.722087 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 01:42:04.722087 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application with Lock( Aug 30 01:42:04.722087 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 01:42:04.722087 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._enter() Aug 30 01:42:04.722087 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 01:42:04.722087 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 01:42:04.722844 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 01:42:04.722844 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.creator() Aug 30 01:42:04.722844 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 01:42:04.722844 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application created_value = creator( Aug 30 01:42:04.722844 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 30 01:42:04.722844 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 30 01:42:04.722844 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 30 01:42:04.722844 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 30 01:42:04.722844 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 30 01:42:04.722844 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 30 01:42:04.722844 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 30 01:42:04.722844 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Aug 30 01:42:04.724973 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 19/34] 10.222.0.27 () {56 vars in 1096 bytes} [Tue Aug 30 01:42:04 2022] GET /identity/v3/roles/admin => generated 83 bytes in 196 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 01:42:04.726452 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c175f303-09c9-4158-9cd4-bfd037012d24 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:42:04.832342 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:42:04.850810 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50fb9c86-9bb8-469a-8773-fb79c6336142 None None] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:42:04.887920 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c175f303-09c9-4158-9cd4-bfd037012d24 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:42:04.898986 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-278f6328-ab6d-4188-aeb2-e5621bc5eb03 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:42:04.899887 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-278f6328-ab6d-4188-aeb2-e5621bc5eb03 None admin] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:04.900350 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-278f6328-ab6d-4188-aeb2-e5621bc5eb03 None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:04.901305 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-278f6328-ab6d-4188-aeb2-e5621bc5eb03 None admin] PATH_INFO: `/v3/projects` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:04.902393 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-278f6328-ab6d-4188-aeb2-e5621bc5eb03 None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:42:04.904167 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-278f6328-ab6d-4188-aeb2-e5621bc5eb03 None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:42:04.927481 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:42:04.939199 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d6da0d1-4a03-43af-b86f-b4aa4f2ed631 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:42:04.939819 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d6da0d1-4a03-43af-b86f-b4aa4f2ed631 None admin] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:04.940096 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d6da0d1-4a03-43af-b86f-b4aa4f2ed631 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:04.940368 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d6da0d1-4a03-43af-b86f-b4aa4f2ed631 None admin] PATH_INFO: `/v3/roles` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:04.941049 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d6da0d1-4a03-43af-b86f-b4aa4f2ed631 None admin] RBAC: Authorizing `identity:create_role()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:42:04.942551 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d6da0d1-4a03-43af-b86f-b4aa4f2ed631 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:42:04.987855 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 16/35] 10.222.0.27 () {60 vars in 1141 bytes} [Tue Aug 30 01:42:04 2022] POST /identity/v3/projects => generated 301 bytes in 298 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 01:42:04.989261 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d53ebfda-5dd6-4ce3-8b7b-5d36f2c8ad6a None None] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:04.989652 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d53ebfda-5dd6-4ce3-8b7b-5d36f2c8ad6a None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:04.989996 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d53ebfda-5dd6-4ce3-8b7b-5d36f2c8ad6a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:05.011804 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.auth.core [None req-d53ebfda-5dd6-4ce3-8b7b-5d36f2c8ad6a None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81578) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 01:42:05.013423 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-d53ebfda-5dd6-4ce3-8b7b-5d36f2c8ad6a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:42:05.101481 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 17/36] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Aug 30 01:42:04 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 30 01:42:05.101481 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d53ebfda-5dd6-4ce3-8b7b-5d36f2c8ad6a None None] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:42:05.116861 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 20/37] 10.222.0.27 () {60 vars in 1131 bytes} [Tue Aug 30 01:42:04 2022] POST /identity/v3/roles => generated 219 bytes in 391 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 01:42:05.118451 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29bd4b11-66bd-4919-b0a5-50299c780a7b None None] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:05.118750 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29bd4b11-66bd-4919-b0a5-50299c780a7b None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:05.119035 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29bd4b11-66bd-4919-b0a5-50299c780a7b None None] PATH_INFO: `/` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:05.121992 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 21/38] 10.222.0.27 () {54 vars in 927 bytes} [Tue Aug 30 01:42:05 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 01:42:05.127935 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44b635fc-390d-4b32-aab0-7d6d95f02f8c None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:05.128263 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44b635fc-390d-4b32-aab0-7d6d95f02f8c None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:05.128512 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44b635fc-390d-4b32-aab0-7d6d95f02f8c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:05.155724 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-44b635fc-390d-4b32-aab0-7d6d95f02f8c None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 01:42:05.157343 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-44b635fc-390d-4b32-aab0-7d6d95f02f8c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:42:05.215003 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:42:05.253170 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 22/39] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Aug 30 01:42:05 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 126 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 30 01:42:05.258409 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d53ebfda-5dd6-4ce3-8b7b-5d36f2c8ad6a None None] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:42:05.299167 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7329a918-32b3-4e8f-a03d-daa0101fc776 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:42:05.299894 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7329a918-32b3-4e8f-a03d-daa0101fc776 None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:05.300075 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7329a918-32b3-4e8f-a03d-daa0101fc776 None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:05.300355 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7329a918-32b3-4e8f-a03d-daa0101fc776 None admin] PATH_INFO: `/v3/roles` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:05.301042 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7329a918-32b3-4e8f-a03d-daa0101fc776 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81578) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:42:05.301354 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7329a918-32b3-4e8f-a03d-daa0101fc776 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:42:05.302886 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7329a918-32b3-4e8f-a03d-daa0101fc776 None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:42:05.315558 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 18/40] 10.222.0.27 () {56 vars in 1099 bytes} [Tue Aug 30 01:42:05 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 223 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:42:05.321427 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44b635fc-390d-4b32-aab0-7d6d95f02f8c None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:42:05.396151 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7329a918-32b3-4e8f-a03d-daa0101fc776 None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:42:05.441728 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44b635fc-390d-4b32-aab0-7d6d95f02f8c None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:42:05.518245 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7329a918-32b3-4e8f-a03d-daa0101fc776 None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:42:05.549655 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:42:05.554517 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-702d88d8-c591-4ce3-ad9c-423848a335db None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:42:05.555137 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-702d88d8-c591-4ce3-ad9c-423848a335db None admin] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:05.555439 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-702d88d8-c591-4ce3-ad9c-423848a335db None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:05.555684 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-702d88d8-c591-4ce3-ad9c-423848a335db None admin] PATH_INFO: `/v3/roles` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:05.556363 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-702d88d8-c591-4ce3-ad9c-423848a335db None admin] RBAC: Authorizing `identity:create_role()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:42:05.558322 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-702d88d8-c591-4ce3-ad9c-423848a335db None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:42:05.564388 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:42:05.584609 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4ba366f-5f18-4659-be50-253d94d7654f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:42:05.585240 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4ba366f-5f18-4659-be50-253d94d7654f None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:05.585552 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4ba366f-5f18-4659-be50-253d94d7654f None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:05.585827 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4ba366f-5f18-4659-be50-253d94d7654f None admin] PATH_INFO: `/v3/users/ac83d5ed111c4f4e8b3c3226bd3bbaba` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:05.586937 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4ba366f-5f18-4659-be50-253d94d7654f None admin] RBAC: Authorizing `identity:get_user(user_id=ac83d5ed111c4f4e8b3c3226bd3bbaba)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:42:05.588543 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4ba366f-5f18-4659-be50-253d94d7654f None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:42:05.590133 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 23/41] 10.222.0.27 () {56 vars in 1177 bytes} [Tue Aug 30 01:42:05 2022] GET /identity/v3/users/ac83d5ed111c4f4e8b3c3226bd3bbaba => generated 247 bytes in 269 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:42:05.590919 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4ba366f-5f18-4659-be50-253d94d7654f None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:42:05.654685 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 19/42] 10.222.0.27 () {60 vars in 1131 bytes} [Tue Aug 30 01:42:05 2022] POST /identity/v3/roles => generated 223 bytes in 259 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 30 01:42:05.655459 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-702d88d8-c591-4ce3-ad9c-423848a335db None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:42:05.699394 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4ba366f-5f18-4659-be50-253d94d7654f None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:42:05.745671 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfbb4a59-10d2-4813-bbba-e22885b76556 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:42:05.746331 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfbb4a59-10d2-4813-bbba-e22885b76556 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:05.746616 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfbb4a59-10d2-4813-bbba-e22885b76556 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:05.746963 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfbb4a59-10d2-4813-bbba-e22885b76556 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:05.748244 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dfbb4a59-10d2-4813-bbba-e22885b76556 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:42:05.749777 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dfbb4a59-10d2-4813-bbba-e22885b76556 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:42:05.753496 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 24/43] 10.222.0.27 () {56 vars in 1108 bytes} [Tue Aug 30 01:42:05 2022] GET /identity/v3/domains/default => generated 199 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:42:05.757284 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfbb4a59-10d2-4813-bbba-e22885b76556 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:42:05.787041 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-702d88d8-c591-4ce3-ad9c-423848a335db None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:42:05.821584 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:42:05.829004 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23deee22-01b9-4b02-8b18-9d473689cd7d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:42:05.829669 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23deee22-01b9-4b02-8b18-9d473689cd7d None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:05.830002 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23deee22-01b9-4b02-8b18-9d473689cd7d None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:05.830277 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23deee22-01b9-4b02-8b18-9d473689cd7d None admin] PATH_INFO: `/v3/domains/default` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:05.831413 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23deee22-01b9-4b02-8b18-9d473689cd7d None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:42:05.832879 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23deee22-01b9-4b02-8b18-9d473689cd7d None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:42:05.834517 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 20/44] 10.222.0.27 () {56 vars in 1108 bytes} [Tue Aug 30 01:42:05 2022] GET /identity/v3/domains/default => generated 199 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:42:05.840623 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23deee22-01b9-4b02-8b18-9d473689cd7d None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:42:05.880986 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfbb4a59-10d2-4813-bbba-e22885b76556 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:42:05.905261 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:42:05.916332 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac809d08-2430-41e9-8ae8-db276b5c256d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:42:05.916946 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac809d08-2430-41e9-8ae8-db276b5c256d None admin] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:05.917616 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac809d08-2430-41e9-8ae8-db276b5c256d None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:05.917616 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac809d08-2430-41e9-8ae8-db276b5c256d None admin] PATH_INFO: `/v3/projects` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:05.918191 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac809d08-2430-41e9-8ae8-db276b5c256d None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:42:05.919643 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac809d08-2430-41e9-8ae8-db276b5c256d None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:42:05.945222 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23deee22-01b9-4b02-8b18-9d473689cd7d None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:42:05.979991 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75949072-5974-4f3a-9003-74c0d1607d88 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:42:05.980652 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75949072-5974-4f3a-9003-74c0d1607d88 None admin] REQUEST_METHOD: `PUT` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:05.980936 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75949072-5974-4f3a-9003-74c0d1607d88 None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:05.981586 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75949072-5974-4f3a-9003-74c0d1607d88 None admin] PATH_INFO: `/v3/domains/default/users/ac83d5ed111c4f4e8b3c3226bd3bbaba/roles/ead5fac86a0747ea8823a82efcea8ed7` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:05.981695 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 25/45] 10.222.0.27 () {60 vars in 1141 bytes} [Tue Aug 30 01:42:05 2022] POST /identity/v3/projects => generated 312 bytes in 225 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 30 01:42:05.983525 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75949072-5974-4f3a-9003-74c0d1607d88 None admin] RBAC: Authorizing `identity:create_grant(domain_id=default, user_id=ac83d5ed111c4f4e8b3c3226bd3bbaba, role_id=ead5fac86a0747ea8823a82efcea8ed7)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:42:05.984996 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75949072-5974-4f3a-9003-74c0d1607d88 None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:42:06.066451 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 21/46] 10.222.0.27 () {58 vars in 1361 bytes} [Tue Aug 30 01:42:05 2022] PUT /identity/v3/domains/default/users/ac83d5ed111c4f4e8b3c3226bd3bbaba/roles/ead5fac86a0747ea8823a82efcea8ed7 => generated 0 bytes in 226 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 30 01:42:07.967438 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e852a7f0-c018-4a6b-bd2c-85d19a9255b1 None None] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:07.967946 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e852a7f0-c018-4a6b-bd2c-85d19a9255b1 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:07.968060 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e852a7f0-c018-4a6b-bd2c-85d19a9255b1 None None] PATH_INFO: `/` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:07.969387 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 26/47] 10.222.0.27 () {54 vars in 927 bytes} [Tue Aug 30 01:42:07 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 01:42:07.975112 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d77fd32e-e572-41d8-be5a-d620aa04d2ee None None] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:07.975505 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d77fd32e-e572-41d8-be5a-d620aa04d2ee None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:07.975606 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d77fd32e-e572-41d8-be5a-d620aa04d2ee None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:07.995356 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.auth.core [None req-d77fd32e-e572-41d8-be5a-d620aa04d2ee None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81578) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 01:42:07.998838 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-d77fd32e-e572-41d8-be5a-d620aa04d2ee None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:42:08.051963 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fff159c0-bb53-4753-b132-6814cb7d31e7 None None] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:08.052483 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fff159c0-bb53-4753-b132-6814cb7d31e7 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:08.052695 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fff159c0-bb53-4753-b132-6814cb7d31e7 None None] PATH_INFO: `/` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:08.054171 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 27/48] 10.222.0.27 () {54 vars in 927 bytes} [Tue Aug 30 01:42:08 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 01:42:08.061598 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2292eec3-ae49-4779-92fe-7ef3ac13a240 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:08.062089 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2292eec3-ae49-4779-92fe-7ef3ac13a240 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:08.062677 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2292eec3-ae49-4779-92fe-7ef3ac13a240 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:08.084013 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-2292eec3-ae49-4779-92fe-7ef3ac13a240 None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 01:42:08.085242 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-2292eec3-ae49-4779-92fe-7ef3ac13a240 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:42:08.092068 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 22/49] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Aug 30 01:42:07 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 118 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Aug 30 01:42:08.153304 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 28/50] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Aug 30 01:42:08 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 30 01:42:08.374806 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d77fd32e-e572-41d8-be5a-d620aa04d2ee None None] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:42:08.424283 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2292eec3-ae49-4779-92fe-7ef3ac13a240 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:42:08.438478 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:42:08.457391 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:42:08.491432 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d77fd32e-e572-41d8-be5a-d620aa04d2ee None None] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:42:08.524632 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b6f66e8-27ee-4821-bd9e-b5a8a3e649ac None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:42:08.525844 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b6f66e8-27ee-4821-bd9e-b5a8a3e649ac None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:08.526240 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b6f66e8-27ee-4821-bd9e-b5a8a3e649ac None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:08.526607 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b6f66e8-27ee-4821-bd9e-b5a8a3e649ac None admin] PATH_INFO: `/v3/domains/default` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:08.527317 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2292eec3-ae49-4779-92fe-7ef3ac13a240 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:42:08.527769 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b6f66e8-27ee-4821-bd9e-b5a8a3e649ac None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:42:08.529522 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b6f66e8-27ee-4821-bd9e-b5a8a3e649ac None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:42:08.531311 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 23/51] 10.222.0.27 () {56 vars in 1108 bytes} [Tue Aug 30 01:42:08 2022] GET /identity/v3/domains/default => generated 199 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:42:08.536806 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b6f66e8-27ee-4821-bd9e-b5a8a3e649ac None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:42:08.563173 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26a261e0-ab59-4b9f-b947-e967507a19d2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:42:08.563810 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26a261e0-ab59-4b9f-b947-e967507a19d2 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:08.564091 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26a261e0-ab59-4b9f-b947-e967507a19d2 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:08.564362 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26a261e0-ab59-4b9f-b947-e967507a19d2 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:08.575014 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26a261e0-ab59-4b9f-b947-e967507a19d2 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 30 01:42:08.579825 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application [None req-26a261e0-ab59-4b9f-b947-e967507a19d2 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 30 01:42:08.579825 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 01:42:08.579825 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 01:42:08.579825 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 01:42:08.579825 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 01:42:08.579825 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 01:42:08.579825 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 01:42:08.579825 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 01:42:08.579825 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 01:42:08.579825 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 01:42:08.579825 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 01:42:08.579825 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 01:42:08.579825 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 30 01:42:08.579825 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 30 01:42:08.579825 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 30 01:42:08.579825 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 30 01:42:08.580752 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 30 01:42:08.580752 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 30 01:42:08.580752 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 30 01:42:08.580752 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 30 01:42:08.580752 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 01:42:08.580752 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 01:42:08.580752 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 01:42:08.580752 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 01:42:08.580752 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 01:42:08.580752 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 01:42:08.580752 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 01:42:08.580752 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application with Lock( Aug 30 01:42:08.580752 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 01:42:08.580752 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._enter() Aug 30 01:42:08.580752 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 01:42:08.580752 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 01:42:08.581562 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 01:42:08.581562 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.creator() Aug 30 01:42:08.581562 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 01:42:08.581562 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application created_value = creator( Aug 30 01:42:08.581562 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 30 01:42:08.581562 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 30 01:42:08.581562 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 30 01:42:08.581562 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 30 01:42:08.581562 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 30 01:42:08.581562 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 30 01:42:08.581562 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 30 01:42:08.581562 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Aug 30 01:42:08.581562 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 29/52] 10.222.0.27 () {56 vars in 1096 bytes} [Tue Aug 30 01:42:08 2022] GET /identity/v3/roles/admin => generated 83 bytes in 157 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 30 01:42:08.588672 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26a261e0-ab59-4b9f-b947-e967507a19d2 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:42:08.685874 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:42:08.694561 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26a261e0-ab59-4b9f-b947-e967507a19d2 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:42:08.740166 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b9dd58c-a12e-4736-bb12-1c7660e5f425 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:42:08.740861 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b9dd58c-a12e-4736-bb12-1c7660e5f425 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:08.741142 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b9dd58c-a12e-4736-bb12-1c7660e5f425 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:08.741416 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b9dd58c-a12e-4736-bb12-1c7660e5f425 None admin] PATH_INFO: `/v3/roles` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:08.742134 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b9dd58c-a12e-4736-bb12-1c7660e5f425 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:42:08.742440 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b9dd58c-a12e-4736-bb12-1c7660e5f425 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:42:08.743861 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b9dd58c-a12e-4736-bb12-1c7660e5f425 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:42:08.749544 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b6f66e8-27ee-4821-bd9e-b5a8a3e649ac None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:42:08.754397 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 30/53] 10.222.0.27 () {56 vars in 1099 bytes} [Tue Aug 30 01:42:08 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 30 01:42:08.760696 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b9dd58c-a12e-4736-bb12-1c7660e5f425 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:42:08.768642 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:42:08.785441 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8a41d57-be48-4e0b-a17a-c7f114d46a3e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:42:08.786090 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8a41d57-be48-4e0b-a17a-c7f114d46a3e None admin] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:08.786367 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8a41d57-be48-4e0b-a17a-c7f114d46a3e None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:08.786636 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8a41d57-be48-4e0b-a17a-c7f114d46a3e None admin] PATH_INFO: `/v3/projects` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:08.787286 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8a41d57-be48-4e0b-a17a-c7f114d46a3e None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:42:08.788729 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8a41d57-be48-4e0b-a17a-c7f114d46a3e None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:42:08.863487 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b9dd58c-a12e-4736-bb12-1c7660e5f425 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:42:08.897082 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3f465f6-a7ba-42dd-8058-fe885413c803 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:42:08.897729 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3f465f6-a7ba-42dd-8058-fe885413c803 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:08.898009 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3f465f6-a7ba-42dd-8058-fe885413c803 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:08.898283 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3f465f6-a7ba-42dd-8058-fe885413c803 None admin] PATH_INFO: `/v3/users/ac83d5ed111c4f4e8b3c3226bd3bbaba` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:08.899452 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3f465f6-a7ba-42dd-8058-fe885413c803 None admin] RBAC: Authorizing `identity:get_user(user_id=ac83d5ed111c4f4e8b3c3226bd3bbaba)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:42:08.901627 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3f465f6-a7ba-42dd-8058-fe885413c803 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:42:08.903205 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 31/54] 10.222.0.27 () {56 vars in 1177 bytes} [Tue Aug 30 01:42:08 2022] GET /identity/v3/users/ac83d5ed111c4f4e8b3c3226bd3bbaba => generated 247 bytes in 144 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 30 01:42:08.908740 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3f465f6-a7ba-42dd-8058-fe885413c803 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:42:08.932686 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:42:08.946858 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 24/55] 10.222.0.27 () {60 vars in 1140 bytes} [Tue Aug 30 01:42:08 2022] POST /identity/v3/projects => generated 298 bytes in 410 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 30 01:42:09.018132 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3f465f6-a7ba-42dd-8058-fe885413c803 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:42:09.056586 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56720889-19b5-441d-8a0b-ea9f803bc898 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:42:09.057693 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56720889-19b5-441d-8a0b-ea9f803bc898 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:09.058146 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56720889-19b5-441d-8a0b-ea9f803bc898 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:09.058427 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56720889-19b5-441d-8a0b-ea9f803bc898 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:09.059674 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56720889-19b5-441d-8a0b-ea9f803bc898 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:42:09.061207 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56720889-19b5-441d-8a0b-ea9f803bc898 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:42:09.062880 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 32/56] 10.222.0.27 () {56 vars in 1108 bytes} [Tue Aug 30 01:42:08 2022] GET /identity/v3/domains/default => generated 199 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 30 01:42:09.069222 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8a41d57-be48-4e0b-a17a-c7f114d46a3e None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:42:09.173640 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8a41d57-be48-4e0b-a17a-c7f114d46a3e None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:42:09.207309 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b95e14b-8ae7-41d2-b505-17cc30e9a24a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:42:09.207843 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b95e14b-8ae7-41d2-b505-17cc30e9a24a None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:09.208123 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b95e14b-8ae7-41d2-b505-17cc30e9a24a None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:09.208394 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b95e14b-8ae7-41d2-b505-17cc30e9a24a None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:09.209124 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b95e14b-8ae7-41d2-b505-17cc30e9a24a None admin] RBAC: Adding query filter params (role.id=ead5fac86a0747ea8823a82efcea8ed7, scope.domain.id=default, user.id=ac83d5ed111c4f4e8b3c3226bd3bbaba) {{(pid=81578) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:42:09.209429 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b95e14b-8ae7-41d2-b505-17cc30e9a24a None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:42:09.210888 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b95e14b-8ae7-41d2-b505-17cc30e9a24a None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:42:09.220014 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 25/57] 10.222.0.27 () {56 vars in 1322 bytes} [Tue Aug 30 01:42:09 2022] GET /identity/v3/role_assignments?user.id=ac83d5ed111c4f4e8b3c3226bd3bbaba&scope.domain.id=default&role.id=ead5fac86a0747ea8823a82efcea8ed7 => generated 530 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:42:10.875001 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60c1a8ba-7c39-47c5-be40-37c05178695f None None] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:10.875545 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60c1a8ba-7c39-47c5-be40-37c05178695f None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:10.875545 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60c1a8ba-7c39-47c5-be40-37c05178695f None None] PATH_INFO: `/` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:10.876938 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 33/58] 10.222.0.27 () {54 vars in 927 bytes} [Tue Aug 30 01:42:10 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 01:42:10.882112 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-354c2121-1fa7-41f4-9db8-a9b13342a9e3 None None] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:10.882523 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-354c2121-1fa7-41f4-9db8-a9b13342a9e3 None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:10.882680 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-354c2121-1fa7-41f4-9db8-a9b13342a9e3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:10.901837 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.auth.core [None req-354c2121-1fa7-41f4-9db8-a9b13342a9e3 None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81578) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 01:42:10.903054 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-354c2121-1fa7-41f4-9db8-a9b13342a9e3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:42:10.929437 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:42:10.969988 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 26/59] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Aug 30 01:42:10 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 30 01:42:11.255480 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60c1a8ba-7c39-47c5-be40-37c05178695f None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:42:11.346717 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:42:11.357270 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60c1a8ba-7c39-47c5-be40-37c05178695f None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:42:11.402493 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebaf6c4b-842c-4796-84ff-f5db19d740ef None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:42:11.403132 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebaf6c4b-842c-4796-84ff-f5db19d740ef None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:11.403416 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebaf6c4b-842c-4796-84ff-f5db19d740ef None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:11.403693 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebaf6c4b-842c-4796-84ff-f5db19d740ef None admin] PATH_INFO: `/v3/domains/default` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:11.404815 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebaf6c4b-842c-4796-84ff-f5db19d740ef None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:42:11.406305 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebaf6c4b-842c-4796-84ff-f5db19d740ef None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:42:11.407873 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 34/60] 10.222.0.27 () {56 vars in 1108 bytes} [Tue Aug 30 01:42:11 2022] GET /identity/v3/domains/default => generated 199 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:42:11.413617 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-354c2121-1fa7-41f4-9db8-a9b13342a9e3 None None] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:42:11.512074 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-354c2121-1fa7-41f4-9db8-a9b13342a9e3 None None] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:42:11.544801 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a432c38a-7d77-4a0d-a7c7-d15bcbe95e46 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:42:11.545405 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a432c38a-7d77-4a0d-a7c7-d15bcbe95e46 None admin] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:11.545711 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a432c38a-7d77-4a0d-a7c7-d15bcbe95e46 None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:11.545981 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a432c38a-7d77-4a0d-a7c7-d15bcbe95e46 None admin] PATH_INFO: `/v3/users` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:11.546634 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a432c38a-7d77-4a0d-a7c7-d15bcbe95e46 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:42:11.548072 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a432c38a-7d77-4a0d-a7c7-d15bcbe95e46 None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:42:11.636640 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 27/61] 10.222.0.27 () {60 vars in 1132 bytes} [Tue Aug 30 01:42:11 2022] POST /identity/v3/users => generated 275 bytes in 224 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 01:42:14.643612 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7d86fee-d628-4154-822c-131fea44f715 None None] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:14.644849 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7d86fee-d628-4154-822c-131fea44f715 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:14.645440 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7d86fee-d628-4154-822c-131fea44f715 None None] PATH_INFO: `/` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:14.647320 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 35/62] 10.222.0.27 () {54 vars in 927 bytes} [Tue Aug 30 01:42:14 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 01:42:14.654251 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-822aaf71-e797-4e34-a61a-c028dd051a63 None None] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:14.655113 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-822aaf71-e797-4e34-a61a-c028dd051a63 None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:14.655596 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-822aaf71-e797-4e34-a61a-c028dd051a63 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:14.679566 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.auth.core [None req-822aaf71-e797-4e34-a61a-c028dd051a63 None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81578) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 01:42:14.681177 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-822aaf71-e797-4e34-a61a-c028dd051a63 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:42:14.699959 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:42:14.781026 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 28/63] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Aug 30 01:42:14 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 128 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 30 01:42:14.805730 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cccf0d18-faf7-4bd4-8801-20c4414f6dd5 None None] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:14.806022 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cccf0d18-faf7-4bd4-8801-20c4414f6dd5 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:14.806317 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cccf0d18-faf7-4bd4-8801-20c4414f6dd5 None None] PATH_INFO: `/` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:14.807781 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 36/64] 10.222.0.27 () {54 vars in 927 bytes} [Tue Aug 30 01:42:14 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 01:42:14.813227 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cfcfc13-a8c7-4b08-aadd-ce3d58151d42 None None] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:14.813558 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cfcfc13-a8c7-4b08-aadd-ce3d58151d42 None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:14.813836 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cfcfc13-a8c7-4b08-aadd-ce3d58151d42 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:14.836889 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.auth.core [None req-8cfcfc13-a8c7-4b08-aadd-ce3d58151d42 None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81578) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 01:42:14.839063 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-8cfcfc13-a8c7-4b08-aadd-ce3d58151d42 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:42:14.925512 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 29/65] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Aug 30 01:42:14 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 113 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 30 01:42:14.938684 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27277900-a96d-46a6-bb13-938cc167ace4 None None] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:14.939237 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27277900-a96d-46a6-bb13-938cc167ace4 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:14.939721 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27277900-a96d-46a6-bb13-938cc167ace4 None None] PATH_INFO: `/` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:14.941562 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 37/66] 10.222.0.27 () {54 vars in 927 bytes} [Tue Aug 30 01:42:14 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 01:42:14.946325 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49620b30-32e9-4a3d-ae47-e8f223352ddb None None] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:14.946618 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49620b30-32e9-4a3d-ae47-e8f223352ddb None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:14.947425 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49620b30-32e9-4a3d-ae47-e8f223352ddb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:14.969107 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.auth.core [None req-49620b30-32e9-4a3d-ae47-e8f223352ddb None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81578) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 01:42:14.970311 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-49620b30-32e9-4a3d-ae47-e8f223352ddb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:42:15.055100 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 30/67] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Aug 30 01:42:14 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 30 01:42:15.088101 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27277900-a96d-46a6-bb13-938cc167ace4 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:42:15.225831 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27277900-a96d-46a6-bb13-938cc167ace4 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:42:15.272250 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7fe4160-d6ae-47b3-aeda-34e49b053d34 None None] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:15.272876 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7fe4160-d6ae-47b3-aeda-34e49b053d34 None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:15.273366 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7fe4160-d6ae-47b3-aeda-34e49b053d34 None None] PATH_INFO: `/` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:15.274015 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8253181b-4ea2-4327-9406-e95c950f7ee8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:42:15.274652 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8253181b-4ea2-4327-9406-e95c950f7ee8 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:15.274931 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8253181b-4ea2-4327-9406-e95c950f7ee8 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:15.275866 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8253181b-4ea2-4327-9406-e95c950f7ee8 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:15.276505 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 31/68] 10.222.0.27 () {54 vars in 927 bytes} [Tue Aug 30 01:42:15 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 30 01:42:15.282215 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9e862c8-82c0-4fe6-80e5-46ddf13356f1 None None] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:15.282733 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9e862c8-82c0-4fe6-80e5-46ddf13356f1 None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:15.283223 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9e862c8-82c0-4fe6-80e5-46ddf13356f1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:15.294581 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8253181b-4ea2-4327-9406-e95c950f7ee8 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 30 01:42:15.303864 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application [None req-8253181b-4ea2-4327-9406-e95c950f7ee8 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 30 01:42:15.303864 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 01:42:15.303864 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 01:42:15.303864 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 01:42:15.303864 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 01:42:15.303864 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 01:42:15.303864 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 01:42:15.303864 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 01:42:15.303864 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 01:42:15.303864 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 01:42:15.303864 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 01:42:15.303864 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 01:42:15.303864 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 30 01:42:15.303864 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 30 01:42:15.303864 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 30 01:42:15.303864 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 30 01:42:15.304726 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 30 01:42:15.304726 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 30 01:42:15.304726 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 30 01:42:15.304726 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 30 01:42:15.304726 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 01:42:15.304726 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 01:42:15.304726 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 01:42:15.304726 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 01:42:15.304726 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 01:42:15.304726 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 01:42:15.304726 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 01:42:15.304726 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application with Lock( Aug 30 01:42:15.304726 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 01:42:15.304726 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._enter() Aug 30 01:42:15.304726 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 01:42:15.304726 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 01:42:15.305509 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 01:42:15.305509 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.creator() Aug 30 01:42:15.305509 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 01:42:15.305509 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application created_value = creator( Aug 30 01:42:15.305509 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 30 01:42:15.305509 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 30 01:42:15.305509 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 30 01:42:15.305509 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 30 01:42:15.305509 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 30 01:42:15.305509 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 30 01:42:15.305509 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 30 01:42:15.305509 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Aug 30 01:42:15.311722 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.auth.core [None req-c9e862c8-82c0-4fe6-80e5-46ddf13356f1 None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81578) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 01:42:15.313062 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 38/69] 10.222.0.27 () {56 vars in 1096 bytes} [Tue Aug 30 01:42:15 2022] GET /identity/v3/roles/admin => generated 83 bytes in 220 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 01:42:15.313946 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8253181b-4ea2-4327-9406-e95c950f7ee8 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:42:15.317191 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-c9e862c8-82c0-4fe6-80e5-46ddf13356f1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:42:15.320880 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:42:15.350490 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:42:15.403167 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 32/70] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Aug 30 01:42:15 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 122 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Aug 30 01:42:15.404322 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9e862c8-82c0-4fe6-80e5-46ddf13356f1 None None] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:42:15.474625 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8253181b-4ea2-4327-9406-e95c950f7ee8 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:42:15.518085 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f39f460c-d024-4603-8071-083ad7e2a59d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:42:15.518624 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f39f460c-d024-4603-8071-083ad7e2a59d None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:15.519231 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f39f460c-d024-4603-8071-083ad7e2a59d None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:15.519521 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f39f460c-d024-4603-8071-083ad7e2a59d None admin] PATH_INFO: `/v3/roles` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:15.520197 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f39f460c-d024-4603-8071-083ad7e2a59d None admin] RBAC: Adding query filter params (name=admin) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:42:15.520509 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f39f460c-d024-4603-8071-083ad7e2a59d None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:42:15.521956 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f39f460c-d024-4603-8071-083ad7e2a59d None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:42:15.522513 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9e862c8-82c0-4fe6-80e5-46ddf13356f1 None None] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:42:15.535483 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 39/71] 10.222.0.27 () {56 vars in 1099 bytes} [Tue Aug 30 01:42:15 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 218 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:42:15.535483 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f39f460c-d024-4603-8071-083ad7e2a59d None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:42:15.560147 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18b2a93f-eca7-4c88-adea-4ac48894d790 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:42:15.560777 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18b2a93f-eca7-4c88-adea-4ac48894d790 None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:15.561062 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18b2a93f-eca7-4c88-adea-4ac48894d790 None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:15.561320 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18b2a93f-eca7-4c88-adea-4ac48894d790 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:15.570345 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:42:15.574216 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18b2a93f-eca7-4c88-adea-4ac48894d790 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 30 01:42:15.580644 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application [None req-18b2a93f-eca7-4c88-adea-4ac48894d790 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Aug 30 01:42:15.580644 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 01:42:15.580644 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 01:42:15.580644 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 01:42:15.580644 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 01:42:15.580644 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 01:42:15.580644 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 01:42:15.580644 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 01:42:15.580644 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 01:42:15.580644 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 01:42:15.580644 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 01:42:15.580644 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 01:42:15.580644 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 30 01:42:15.580644 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 30 01:42:15.580644 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 30 01:42:15.580644 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 30 01:42:15.587282 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 30 01:42:15.587282 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 30 01:42:15.587282 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 30 01:42:15.587282 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 30 01:42:15.587282 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 01:42:15.587282 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 01:42:15.587282 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 01:42:15.587282 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 01:42:15.587282 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 01:42:15.587282 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 01:42:15.587282 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 01:42:15.587282 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application with Lock( Aug 30 01:42:15.587282 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 01:42:15.587282 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._enter() Aug 30 01:42:15.587282 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 01:42:15.587282 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 01:42:15.588480 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 01:42:15.588480 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.creator() Aug 30 01:42:15.588480 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 01:42:15.588480 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application created_value = creator( Aug 30 01:42:15.588480 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 30 01:42:15.588480 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 30 01:42:15.588480 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 30 01:42:15.588480 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 30 01:42:15.588480 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 30 01:42:15.588480 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 30 01:42:15.588480 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Aug 30 01:42:15.588480 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application Aug 30 01:42:15.588480 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 33/72] 10.222.0.27 () {56 vars in 1114 bytes} [Tue Aug 30 01:42:15 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 178 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 01:42:15.588480 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68ae10a1-3c34-44dd-bf5c-2c635e4deddc None None] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:15.589171 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68ae10a1-3c34-44dd-bf5c-2c635e4deddc None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:15.589171 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68ae10a1-3c34-44dd-bf5c-2c635e4deddc None None] PATH_INFO: `/` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:15.589171 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 34/73] 10.222.0.27 () {54 vars in 927 bytes} [Tue Aug 30 01:42:15 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 01:42:15.589171 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68ae10a1-3c34-44dd-bf5c-2c635e4deddc None None] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:42:15.631038 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:42:15.653033 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f39f460c-d024-4603-8071-083ad7e2a59d None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:42:15.688148 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3898784c-4d44-431a-bbf0-295e3beaad97 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:42:15.688783 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3898784c-4d44-431a-bbf0-295e3beaad97 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:15.689062 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3898784c-4d44-431a-bbf0-295e3beaad97 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:15.689333 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3898784c-4d44-431a-bbf0-295e3beaad97 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:15.700099 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3898784c-4d44-431a-bbf0-295e3beaad97 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 30 01:42:15.705349 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application [None req-3898784c-4d44-431a-bbf0-295e3beaad97 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Aug 30 01:42:15.705349 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 01:42:15.705349 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 01:42:15.705349 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 01:42:15.705349 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 01:42:15.705349 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 01:42:15.705349 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 01:42:15.705349 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 01:42:15.705349 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 01:42:15.705349 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 01:42:15.705349 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 01:42:15.705349 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 01:42:15.705349 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 30 01:42:15.705349 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 30 01:42:15.705349 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 30 01:42:15.705349 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 30 01:42:15.706452 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 30 01:42:15.706452 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 30 01:42:15.706452 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 30 01:42:15.706452 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 30 01:42:15.706452 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 01:42:15.706452 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 01:42:15.706452 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 01:42:15.706452 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 01:42:15.706452 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 01:42:15.706452 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 01:42:15.706452 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 01:42:15.706452 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application with Lock( Aug 30 01:42:15.706452 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 01:42:15.706452 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._enter() Aug 30 01:42:15.706452 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 01:42:15.706452 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 01:42:15.710375 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 01:42:15.710375 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.creator() Aug 30 01:42:15.710375 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 01:42:15.710375 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application created_value = creator( Aug 30 01:42:15.710375 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 30 01:42:15.710375 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 30 01:42:15.710375 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 30 01:42:15.710375 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 30 01:42:15.710375 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 30 01:42:15.710375 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 30 01:42:15.710375 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Aug 30 01:42:15.710375 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Aug 30 01:42:15.710959 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68ae10a1-3c34-44dd-bf5c-2c635e4deddc None None] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:42:15.711348 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 40/74] 10.222.0.27 () {56 vars in 1099 bytes} [Tue Aug 30 01:42:15 2022] GET /identity/v3/roles/member => generated 84 bytes in 175 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 01:42:15.711348 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3898784c-4d44-431a-bbf0-295e3beaad97 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:42:15.744829 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58e547ed-e973-48de-86e1-9c045a11bd33 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:42:15.745484 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58e547ed-e973-48de-86e1-9c045a11bd33 None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:15.745766 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58e547ed-e973-48de-86e1-9c045a11bd33 None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:15.746040 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58e547ed-e973-48de-86e1-9c045a11bd33 None admin] PATH_INFO: `/v3/users/ac83d5ed111c4f4e8b3c3226bd3bbaba` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:15.747180 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58e547ed-e973-48de-86e1-9c045a11bd33 None admin] RBAC: Authorizing `identity:get_user(user_id=ac83d5ed111c4f4e8b3c3226bd3bbaba)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:42:15.748645 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58e547ed-e973-48de-86e1-9c045a11bd33 None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:42:15.750261 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 35/75] 10.222.0.27 () {56 vars in 1177 bytes} [Tue Aug 30 01:42:15 2022] GET /identity/v3/users/ac83d5ed111c4f4e8b3c3226bd3bbaba => generated 247 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:42:15.751373 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a9c497c-dac6-472b-a5cf-c71f37a66726 None None] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:15.751657 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a9c497c-dac6-472b-a5cf-c71f37a66726 None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:15.751929 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a9c497c-dac6-472b-a5cf-c71f37a66726 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:15.769900 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.auth.core [None req-8a9c497c-dac6-472b-a5cf-c71f37a66726 None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81578) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 01:42:15.771020 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-8a9c497c-dac6-472b-a5cf-c71f37a66726 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:42:15.813173 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3898784c-4d44-431a-bbf0-295e3beaad97 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:42:15.832886 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:42:15.838684 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 36/76] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Aug 30 01:42:15 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 30 01:42:15.839588 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a9c497c-dac6-472b-a5cf-c71f37a66726 None None] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:42:15.848700 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e810f5d-0844-4df3-b478-3518d7b5f77f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:42:15.849545 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e810f5d-0844-4df3-b478-3518d7b5f77f None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:15.873634 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e810f5d-0844-4df3-b478-3518d7b5f77f None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:15.873634 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e810f5d-0844-4df3-b478-3518d7b5f77f None admin] PATH_INFO: `/v3/roles` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:15.873634 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e810f5d-0844-4df3-b478-3518d7b5f77f None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:42:15.873634 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e810f5d-0844-4df3-b478-3518d7b5f77f None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:42:15.873634 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e810f5d-0844-4df3-b478-3518d7b5f77f None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:42:15.873634 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 41/77] 10.222.0.27 () {56 vars in 1111 bytes} [Tue Aug 30 01:42:15 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:42:15.874104 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e810f5d-0844-4df3-b478-3518d7b5f77f None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:42:15.874104 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:42:15.949598 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a9c497c-dac6-472b-a5cf-c71f37a66726 None None] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:42:15.975064 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e810f5d-0844-4df3-b478-3518d7b5f77f None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:42:15.986997 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eba1f62b-81b0-4941-9f18-509fa1d145e8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:42:15.987634 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eba1f62b-81b0-4941-9f18-509fa1d145e8 None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:15.987916 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eba1f62b-81b0-4941-9f18-509fa1d145e8 None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:15.988190 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eba1f62b-81b0-4941-9f18-509fa1d145e8 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:16.001110 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eba1f62b-81b0-4941-9f18-509fa1d145e8 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 30 01:42:16.006504 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application [None req-eba1f62b-81b0-4941-9f18-509fa1d145e8 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Aug 30 01:42:16.006504 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 01:42:16.006504 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 01:42:16.006504 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 01:42:16.006504 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 01:42:16.006504 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 01:42:16.006504 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 01:42:16.006504 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 01:42:16.006504 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 01:42:16.006504 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 01:42:16.006504 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 01:42:16.006504 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 01:42:16.006504 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 30 01:42:16.006504 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 30 01:42:16.006504 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 30 01:42:16.006504 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 30 01:42:16.007340 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 30 01:42:16.007340 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 30 01:42:16.007340 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 30 01:42:16.007340 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 30 01:42:16.007340 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 01:42:16.007340 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 01:42:16.007340 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 01:42:16.007340 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 01:42:16.007340 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 01:42:16.007340 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 01:42:16.007340 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 01:42:16.007340 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application with Lock( Aug 30 01:42:16.007340 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 01:42:16.007340 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._enter() Aug 30 01:42:16.007340 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 01:42:16.007340 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 01:42:16.008121 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 01:42:16.008121 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.creator() Aug 30 01:42:16.008121 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 01:42:16.008121 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application created_value = creator( Aug 30 01:42:16.008121 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 30 01:42:16.008121 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 30 01:42:16.008121 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 30 01:42:16.008121 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 30 01:42:16.008121 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 30 01:42:16.008121 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 30 01:42:16.008121 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Aug 30 01:42:16.008121 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application Aug 30 01:42:16.008121 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 37/78] 10.222.0.27 () {56 vars in 1099 bytes} [Tue Aug 30 01:42:15 2022] GET /identity/v3/roles/member => generated 84 bytes in 168 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 01:42:16.008756 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eba1f62b-81b0-4941-9f18-509fa1d145e8 None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:42:16.016030 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45c57921-696b-453e-9367-2a7aad716883 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:42:16.016644 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45c57921-696b-453e-9367-2a7aad716883 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:16.016927 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45c57921-696b-453e-9367-2a7aad716883 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:16.017197 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45c57921-696b-453e-9367-2a7aad716883 None admin] PATH_INFO: `/v3/roles` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:16.017889 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45c57921-696b-453e-9367-2a7aad716883 None admin] RBAC: Adding query filter params (name=member) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:42:16.018193 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45c57921-696b-453e-9367-2a7aad716883 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:42:16.019579 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45c57921-696b-453e-9367-2a7aad716883 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:42:16.028624 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 42/79] 10.222.0.27 () {56 vars in 1101 bytes} [Tue Aug 30 01:42:15 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:42:16.029384 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45c57921-696b-453e-9367-2a7aad716883 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:42:16.075996 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:42:16.099021 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:42:16.117783 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eba1f62b-81b0-4941-9f18-509fa1d145e8 None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:42:16.139512 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45c57921-696b-453e-9367-2a7aad716883 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:42:16.158884 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-958f048d-8b64-4e23-adc7-28361c576990 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:42:16.159493 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-958f048d-8b64-4e23-adc7-28361c576990 None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:16.159779 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-958f048d-8b64-4e23-adc7-28361c576990 None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:16.160051 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-958f048d-8b64-4e23-adc7-28361c576990 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:16.175084 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1803a98c-8ea2-4329-b0b3-38f64bfcfe40 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:42:16.175741 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1803a98c-8ea2-4329-b0b3-38f64bfcfe40 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:16.176022 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1803a98c-8ea2-4329-b0b3-38f64bfcfe40 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:16.176301 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1803a98c-8ea2-4329-b0b3-38f64bfcfe40 None admin] PATH_INFO: `/v3/users/fac663577ae547588ada9d16b2fd6f5f` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:16.190358 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1803a98c-8ea2-4329-b0b3-38f64bfcfe40 None admin] RBAC: Authorizing `identity:get_user(user_id=fac663577ae547588ada9d16b2fd6f5f)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:42:16.191830 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1803a98c-8ea2-4329-b0b3-38f64bfcfe40 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:42:16.193519 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 43/80] 10.222.0.27 () {56 vars in 1177 bytes} [Tue Aug 30 01:42:16 2022] GET /identity/v3/users/fac663577ae547588ada9d16b2fd6f5f => generated 275 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:42:16.194290 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1803a98c-8ea2-4329-b0b3-38f64bfcfe40 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:42:16.234132 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-958f048d-8b64-4e23-adc7-28361c576990 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:42:16.235639 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-958f048d-8b64-4e23-adc7-28361c576990 None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:42:16.292196 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:42:16.293408 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1803a98c-8ea2-4329-b0b3-38f64bfcfe40 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:42:16.316847 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:42:16.329081 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d40f0e31-8271-4400-b6f2-0575ec39065e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:42:16.329715 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d40f0e31-8271-4400-b6f2-0575ec39065e None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:16.329996 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d40f0e31-8271-4400-b6f2-0575ec39065e None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:16.330267 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d40f0e31-8271-4400-b6f2-0575ec39065e None admin] PATH_INFO: `/v3/roles` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:16.330954 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d40f0e31-8271-4400-b6f2-0575ec39065e None admin] RBAC: Adding query filter params (name=member) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:42:16.331270 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d40f0e31-8271-4400-b6f2-0575ec39065e None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:42:16.332638 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d40f0e31-8271-4400-b6f2-0575ec39065e None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:42:16.334715 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 38/81] 10.222.0.27 () {58 vars in 1282 bytes} [Tue Aug 30 01:42:16 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 327 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 30 01:42:16.335558 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-958f048d-8b64-4e23-adc7-28361c576990 None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:42:16.343946 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 44/82] 10.222.0.27 () {56 vars in 1101 bytes} [Tue Aug 30 01:42:16 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:42:16.345055 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d40f0e31-8271-4400-b6f2-0575ec39065e None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:42:16.437569 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-958f048d-8b64-4e23-adc7-28361c576990 None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:42:16.449004 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d40f0e31-8271-4400-b6f2-0575ec39065e None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:42:16.470889 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be608c23-2f87-42d4-b57e-74186dfd9325 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:42:16.471518 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be608c23-2f87-42d4-b57e-74186dfd9325 None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:16.471793 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be608c23-2f87-42d4-b57e-74186dfd9325 None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:16.472061 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be608c23-2f87-42d4-b57e-74186dfd9325 None admin] PATH_INFO: `/v3/users/fac663577ae547588ada9d16b2fd6f5f` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:16.473174 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be608c23-2f87-42d4-b57e-74186dfd9325 None admin] RBAC: Authorizing `identity:get_user(user_id=fac663577ae547588ada9d16b2fd6f5f)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:42:16.474649 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be608c23-2f87-42d4-b57e-74186dfd9325 None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:42:16.476481 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 39/83] 10.222.0.27 () {56 vars in 1177 bytes} [Tue Aug 30 01:42:16 2022] GET /identity/v3/users/fac663577ae547588ada9d16b2fd6f5f => generated 275 bytes in 141 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:42:16.477234 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be608c23-2f87-42d4-b57e-74186dfd9325 None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:42:16.488105 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e4da245-52e7-44ed-a828-6d7e7946e0fc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:42:16.488774 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e4da245-52e7-44ed-a828-6d7e7946e0fc None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:16.489059 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e4da245-52e7-44ed-a828-6d7e7946e0fc None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:16.489331 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e4da245-52e7-44ed-a828-6d7e7946e0fc None admin] PATH_INFO: `/v3/domains/default` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:16.490488 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e4da245-52e7-44ed-a828-6d7e7946e0fc None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:42:16.491992 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e4da245-52e7-44ed-a828-6d7e7946e0fc None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:42:16.493795 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 45/84] 10.222.0.27 () {56 vars in 1108 bytes} [Tue Aug 30 01:42:16 2022] GET /identity/v3/domains/default => generated 199 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:42:16.495257 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e4da245-52e7-44ed-a828-6d7e7946e0fc None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:42:16.510622 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:42:16.547873 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:42:16.583500 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be608c23-2f87-42d4-b57e-74186dfd9325 None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:42:16.596915 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e4da245-52e7-44ed-a828-6d7e7946e0fc None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:42:16.619548 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae6c8cea-dc0b-403a-8b8a-15530f8006d4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:42:16.620299 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae6c8cea-dc0b-403a-8b8a-15530f8006d4 None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:16.620574 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae6c8cea-dc0b-403a-8b8a-15530f8006d4 None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:16.620842 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae6c8cea-dc0b-403a-8b8a-15530f8006d4 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:16.631591 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be71914b-511f-4622-817e-5a59acc093bf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:42:16.632206 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be71914b-511f-4622-817e-5a59acc093bf None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:16.632496 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be71914b-511f-4622-817e-5a59acc093bf None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:16.632765 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be71914b-511f-4622-817e-5a59acc093bf None admin] PATH_INFO: `/v3/projects/90818f998d5f4520971eacf6ccc830ce` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:16.633917 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be71914b-511f-4622-817e-5a59acc093bf None admin] RBAC: Authorizing `identity:get_project(project_id=90818f998d5f4520971eacf6ccc830ce)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:42:16.636026 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be71914b-511f-4622-817e-5a59acc093bf None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:42:16.637798 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 46/85] 10.222.0.27 () {56 vars in 1186 bytes} [Tue Aug 30 01:42:16 2022] GET /identity/v3/projects/90818f998d5f4520971eacf6ccc830ce => generated 298 bytes in 144 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:42:16.638559 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be71914b-511f-4622-817e-5a59acc093bf None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:42:16.694726 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae6c8cea-dc0b-403a-8b8a-15530f8006d4 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:42:16.696186 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae6c8cea-dc0b-403a-8b8a-15530f8006d4 None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:42:16.722429 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:42:16.741373 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be71914b-511f-4622-817e-5a59acc093bf None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:42:16.762493 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:42:16.777130 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3270b6e6-4d06-4034-b4cb-2f956fe7ec3b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:42:16.777802 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3270b6e6-4d06-4034-b4cb-2f956fe7ec3b None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:16.778085 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3270b6e6-4d06-4034-b4cb-2f956fe7ec3b None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:16.778363 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3270b6e6-4d06-4034-b4cb-2f956fe7ec3b None admin] PATH_INFO: `/v3/users/fac663577ae547588ada9d16b2fd6f5f` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:16.779667 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3270b6e6-4d06-4034-b4cb-2f956fe7ec3b None admin] RBAC: Authorizing `identity:get_user(user_id=fac663577ae547588ada9d16b2fd6f5f)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:42:16.781318 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3270b6e6-4d06-4034-b4cb-2f956fe7ec3b None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:42:16.783233 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 47/86] 10.222.0.27 () {56 vars in 1177 bytes} [Tue Aug 30 01:42:16 2022] GET /identity/v3/users/fac663577ae547588ada9d16b2fd6f5f => generated 275 bytes in 145 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:42:16.784000 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3270b6e6-4d06-4034-b4cb-2f956fe7ec3b None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:42:16.800572 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 40/87] 10.222.0.27 () {58 vars in 1282 bytes} [Tue Aug 30 01:42:16 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 323 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 30 01:42:16.801336 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae6c8cea-dc0b-403a-8b8a-15530f8006d4 None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:42:16.883545 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3270b6e6-4d06-4034-b4cb-2f956fe7ec3b None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:42:16.902229 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae6c8cea-dc0b-403a-8b8a-15530f8006d4 None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:42:16.918008 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3da2869f-1928-44b2-a943-573479589428 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:42:16.918621 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3da2869f-1928-44b2-a943-573479589428 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:16.918903 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3da2869f-1928-44b2-a943-573479589428 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:16.919173 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3da2869f-1928-44b2-a943-573479589428 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:16.928229 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:42:16.937914 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bed11bb-3829-4b46-af12-a4e2c8c728a2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:42:16.938529 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bed11bb-3829-4b46-af12-a4e2c8c728a2 None admin] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:16.938808 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bed11bb-3829-4b46-af12-a4e2c8c728a2 None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:16.939086 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bed11bb-3829-4b46-af12-a4e2c8c728a2 None admin] PATH_INFO: `/v3/users` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:16.940604 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bed11bb-3829-4b46-af12-a4e2c8c728a2 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:42:16.941153 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bed11bb-3829-4b46-af12-a4e2c8c728a2 None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:42:16.977609 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:42:16.989603 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3da2869f-1928-44b2-a943-573479589428 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:42:16.991107 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3da2869f-1928-44b2-a943-573479589428 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:42:17.003012 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 41/88] 10.222.0.27 () {60 vars in 1132 bytes} [Tue Aug 30 01:42:16 2022] POST /identity/v3/users => generated 289 bytes in 202 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 01:42:17.003838 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bed11bb-3829-4b46-af12-a4e2c8c728a2 None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:42:17.116685 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 48/89] 10.222.0.27 () {58 vars in 1282 bytes} [Tue Aug 30 01:42:16 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 333 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 30 01:42:17.117634 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3da2869f-1928-44b2-a943-573479589428 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:42:17.118537 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bed11bb-3829-4b46-af12-a4e2c8c728a2 None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:42:17.162491 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cae4b96e-db68-48f9-9c62-fc2512b10ef4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:42:17.163348 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cae4b96e-db68-48f9-9c62-fc2512b10ef4 None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:17.163639 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cae4b96e-db68-48f9-9c62-fc2512b10ef4 None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:17.163918 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cae4b96e-db68-48f9-9c62-fc2512b10ef4 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:17.164661 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cae4b96e-db68-48f9-9c62-fc2512b10ef4 None admin] RBAC: Adding query filter params (role.id=ead5fac86a0747ea8823a82efcea8ed7, scope.project.id=90818f998d5f4520971eacf6ccc830ce, user.id=ac83d5ed111c4f4e8b3c3226bd3bbaba) {{(pid=81578) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:42:17.164972 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cae4b96e-db68-48f9-9c62-fc2512b10ef4 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:42:17.167042 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cae4b96e-db68-48f9-9c62-fc2512b10ef4 None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:42:17.173022 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 42/90] 10.222.0.27 () {56 vars in 1374 bytes} [Tue Aug 30 01:42:17 2022] GET /identity/v3/role_assignments?user.id=ac83d5ed111c4f4e8b3c3226bd3bbaba&scope.project.id=90818f998d5f4520971eacf6ccc830ce&role.id=ead5fac86a0747ea8823a82efcea8ed7 => generated 259 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:42:17.173884 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cae4b96e-db68-48f9-9c62-fc2512b10ef4 None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:42:17.240227 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:42:17.244740 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:42:17.264572 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3da2869f-1928-44b2-a943-573479589428 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:42:17.313930 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-109db58a-a88a-4665-8525-b99d09514fe4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:42:17.314985 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-109db58a-a88a-4665-8525-b99d09514fe4 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:17.314985 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-109db58a-a88a-4665-8525-b99d09514fe4 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:17.315150 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-109db58a-a88a-4665-8525-b99d09514fe4 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:17.322053 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cae4b96e-db68-48f9-9c62-fc2512b10ef4 None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:42:17.373981 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65c0ca75-13bf-4f7b-b437-050d905f33c1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:42:17.374549 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65c0ca75-13bf-4f7b-b437-050d905f33c1 None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:17.374835 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65c0ca75-13bf-4f7b-b437-050d905f33c1 None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:17.375113 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65c0ca75-13bf-4f7b-b437-050d905f33c1 None admin] PATH_INFO: `/v3/projects/90818f998d5f4520971eacf6ccc830ce` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:17.376359 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65c0ca75-13bf-4f7b-b437-050d905f33c1 None admin] RBAC: Authorizing `identity:get_project(project_id=90818f998d5f4520971eacf6ccc830ce)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:42:17.378183 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65c0ca75-13bf-4f7b-b437-050d905f33c1 None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:42:17.379897 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 43/91] 10.222.0.27 () {56 vars in 1186 bytes} [Tue Aug 30 01:42:17 2022] GET /identity/v3/projects/90818f998d5f4520971eacf6ccc830ce => generated 298 bytes in 206 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:42:17.380987 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65c0ca75-13bf-4f7b-b437-050d905f33c1 None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:42:17.407686 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-109db58a-a88a-4665-8525-b99d09514fe4 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:42:17.409335 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-109db58a-a88a-4665-8525-b99d09514fe4 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:42:17.524028 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:42:17.527926 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65c0ca75-13bf-4f7b-b437-050d905f33c1 None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:42:17.543721 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 49/92] 10.222.0.27 () {58 vars in 1282 bytes} [Tue Aug 30 01:42:17 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 427 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 30 01:42:17.545117 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-109db58a-a88a-4665-8525-b99d09514fe4 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:42:17.549493 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:42:17.575802 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1834db75-7aac-4165-9b16-9606c26b5949 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:42:17.576458 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1834db75-7aac-4165-9b16-9606c26b5949 None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:17.576746 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1834db75-7aac-4165-9b16-9606c26b5949 None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:17.577025 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1834db75-7aac-4165-9b16-9606c26b5949 None admin] PATH_INFO: `/v3/projects/90818f998d5f4520971eacf6ccc830ce` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:17.578302 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1834db75-7aac-4165-9b16-9606c26b5949 None admin] RBAC: Authorizing `identity:get_project(project_id=90818f998d5f4520971eacf6ccc830ce)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:42:17.579768 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1834db75-7aac-4165-9b16-9606c26b5949 None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:42:17.581567 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 44/93] 10.222.0.27 () {56 vars in 1186 bytes} [Tue Aug 30 01:42:17 2022] GET /identity/v3/projects/90818f998d5f4520971eacf6ccc830ce => generated 298 bytes in 201 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:42:17.582341 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1834db75-7aac-4165-9b16-9606c26b5949 None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:42:17.680905 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-109db58a-a88a-4665-8525-b99d09514fe4 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:42:17.732194 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f91ad22e-5b6d-48f3-8045-7b164ff71972 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:42:17.732826 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f91ad22e-5b6d-48f3-8045-7b164ff71972 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:17.733106 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f91ad22e-5b6d-48f3-8045-7b164ff71972 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:17.733378 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f91ad22e-5b6d-48f3-8045-7b164ff71972 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:17.734141 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f91ad22e-5b6d-48f3-8045-7b164ff71972 None admin] RBAC: Adding query filter params (role.id=4621068972ba4f79bad104f956db04aa, scope.project.id=90818f998d5f4520971eacf6ccc830ce, user.id=fac663577ae547588ada9d16b2fd6f5f) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:42:17.734451 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f91ad22e-5b6d-48f3-8045-7b164ff71972 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:42:17.735909 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f91ad22e-5b6d-48f3-8045-7b164ff71972 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:42:17.751072 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 50/94] 10.222.0.27 () {56 vars in 1374 bytes} [Tue Aug 30 01:42:17 2022] GET /identity/v3/role_assignments?user.id=fac663577ae547588ada9d16b2fd6f5f&scope.project.id=90818f998d5f4520971eacf6ccc830ce&role.id=4621068972ba4f79bad104f956db04aa => generated 259 bytes in 206 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:42:17.751936 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f91ad22e-5b6d-48f3-8045-7b164ff71972 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:42:17.787724 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1834db75-7aac-4165-9b16-9606c26b5949 None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:42:17.874022 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:42:17.925570 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a4dfc5e-fdb0-4278-8655-9c8c216aa012 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:42:17.926304 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a4dfc5e-fdb0-4278-8655-9c8c216aa012 None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:17.926594 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a4dfc5e-fdb0-4278-8655-9c8c216aa012 None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:17.926874 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a4dfc5e-fdb0-4278-8655-9c8c216aa012 None admin] PATH_INFO: `/v3/projects/2b886b3234b7472ca4968ccca426a61d` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:17.928177 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a4dfc5e-fdb0-4278-8655-9c8c216aa012 None admin] RBAC: Authorizing `identity:get_project(project_id=2b886b3234b7472ca4968ccca426a61d)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:42:17.929692 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a4dfc5e-fdb0-4278-8655-9c8c216aa012 None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:42:17.932032 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 45/95] 10.222.0.27 () {56 vars in 1186 bytes} [Tue Aug 30 01:42:17 2022] GET /identity/v3/projects/2b886b3234b7472ca4968ccca426a61d => generated 312 bytes in 350 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:42:17.940021 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a4dfc5e-fdb0-4278-8655-9c8c216aa012 None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:42:17.953626 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f91ad22e-5b6d-48f3-8045-7b164ff71972 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:42:18.046194 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66ee5487-5056-47de-a929-c7716b6afabc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:42:18.047184 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66ee5487-5056-47de-a929-c7716b6afabc None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:18.047686 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66ee5487-5056-47de-a929-c7716b6afabc None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:18.048158 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66ee5487-5056-47de-a929-c7716b6afabc None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:18.049108 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66ee5487-5056-47de-a929-c7716b6afabc None admin] RBAC: Adding query filter params (role.id=1fb81afcb9f245f3b989add94598a063, scope.project.id=90818f998d5f4520971eacf6ccc830ce, user.id=fac663577ae547588ada9d16b2fd6f5f) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:42:18.049648 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66ee5487-5056-47de-a929-c7716b6afabc None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:42:18.051308 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66ee5487-5056-47de-a929-c7716b6afabc None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:42:18.061403 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 51/96] 10.222.0.27 () {56 vars in 1374 bytes} [Tue Aug 30 01:42:17 2022] GET /identity/v3/role_assignments?user.id=fac663577ae547588ada9d16b2fd6f5f&scope.project.id=90818f998d5f4520971eacf6ccc830ce&role.id=1fb81afcb9f245f3b989add94598a063 => generated 259 bytes in 310 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:42:18.067695 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:42:18.152726 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a4dfc5e-fdb0-4278-8655-9c8c216aa012 None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:42:18.204353 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a194bff-b33b-43ea-bd3e-10437c23351f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:42:18.204891 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a194bff-b33b-43ea-bd3e-10437c23351f None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:18.205177 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a194bff-b33b-43ea-bd3e-10437c23351f None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:18.205470 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a194bff-b33b-43ea-bd3e-10437c23351f None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:18.206226 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a194bff-b33b-43ea-bd3e-10437c23351f None admin] RBAC: Adding query filter params (role.id=1fb81afcb9f245f3b989add94598a063, scope.project.id=2b886b3234b7472ca4968ccca426a61d, user.id=fac663577ae547588ada9d16b2fd6f5f) {{(pid=81578) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:42:18.206540 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a194bff-b33b-43ea-bd3e-10437c23351f None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:42:18.207980 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a194bff-b33b-43ea-bd3e-10437c23351f None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:42:18.214300 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 46/97] 10.222.0.27 () {56 vars in 1374 bytes} [Tue Aug 30 01:42:17 2022] GET /identity/v3/role_assignments?user.id=fac663577ae547588ada9d16b2fd6f5f&scope.project.id=2b886b3234b7472ca4968ccca426a61d&role.id=1fb81afcb9f245f3b989add94598a063 => generated 259 bytes in 275 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:42:19.687412 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-746100cf-c35e-4d0a-b64a-a613aedb2e2e None None] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:19.688471 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-746100cf-c35e-4d0a-b64a-a613aedb2e2e None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:19.688997 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-746100cf-c35e-4d0a-b64a-a613aedb2e2e None None] PATH_INFO: `/` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:19.690633 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 52/98] 10.222.0.27 () {54 vars in 927 bytes} [Tue Aug 30 01:42:19 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 01:42:19.707324 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-536a2e79-2c94-42ad-a393-bd690b6a420b None None] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:19.708233 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-536a2e79-2c94-42ad-a393-bd690b6a420b None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:19.708709 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-536a2e79-2c94-42ad-a393-bd690b6a420b None None] PATH_INFO: `/` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:19.715093 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 47/99] 10.222.0.27 () {54 vars in 927 bytes} [Tue Aug 30 01:42:19 2022] GET /identity => generated 270 bytes in 14 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 01:42:19.722236 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d4099dc-bd45-4cf5-9c52-459ea002711c None None] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:19.722996 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-703b0a41-2346-41fc-bf35-018e407f61ec None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:19.723480 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-703b0a41-2346-41fc-bf35-018e407f61ec None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:19.723951 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d4099dc-bd45-4cf5-9c52-459ea002711c None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:19.724450 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-703b0a41-2346-41fc-bf35-018e407f61ec None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:19.724913 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d4099dc-bd45-4cf5-9c52-459ea002711c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:19.765900 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.auth.core [None req-8d4099dc-bd45-4cf5-9c52-459ea002711c None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81578) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 01:42:19.770908 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-703b0a41-2346-41fc-bf35-018e407f61ec None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 01:42:19.771634 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-8d4099dc-bd45-4cf5-9c52-459ea002711c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:42:19.772996 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-703b0a41-2346-41fc-bf35-018e407f61ec None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:42:19.896519 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 48/100] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Aug 30 01:42:19 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 181 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 30 01:42:19.898824 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 53/101] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Aug 30 01:42:19 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 188 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 30 01:42:20.066323 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5f04182-0e3e-4b82-b3b9-a7b24835fe4e None None] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:20.066660 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5f04182-0e3e-4b82-b3b9-a7b24835fe4e None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:20.066874 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5f04182-0e3e-4b82-b3b9-a7b24835fe4e None None] PATH_INFO: `/` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:20.068218 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 49/102] 10.222.0.27 () {54 vars in 927 bytes} [Tue Aug 30 01:42:20 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 01:42:20.074769 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ea833c2-89ac-4353-a948-05a148a0cccf None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:20.075279 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ea833c2-89ac-4353-a948-05a148a0cccf None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:20.075768 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ea833c2-89ac-4353-a948-05a148a0cccf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:20.098150 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-2ea833c2-89ac-4353-a948-05a148a0cccf None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 01:42:20.099649 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-2ea833c2-89ac-4353-a948-05a148a0cccf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:42:20.154220 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:42:20.179883 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 54/103] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Aug 30 01:42:20 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 30 01:42:20.264189 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5f04182-0e3e-4b82-b3b9-a7b24835fe4e None None] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:42:20.327042 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:42:20.391717 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5f04182-0e3e-4b82-b3b9-a7b24835fe4e None None] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:42:20.435103 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55380345-ee11-406f-94e1-ab0059afb019 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:42:20.436088 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55380345-ee11-406f-94e1-ab0059afb019 None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:20.436586 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55380345-ee11-406f-94e1-ab0059afb019 None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:20.437069 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55380345-ee11-406f-94e1-ab0059afb019 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:20.452167 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55380345-ee11-406f-94e1-ab0059afb019 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 30 01:42:20.458169 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application [None req-55380345-ee11-406f-94e1-ab0059afb019 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 30 01:42:20.458169 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 01:42:20.458169 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 01:42:20.458169 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 01:42:20.458169 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 01:42:20.458169 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 01:42:20.458169 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 01:42:20.458169 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 01:42:20.458169 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 01:42:20.458169 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 01:42:20.458169 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 01:42:20.458169 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 01:42:20.458169 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 30 01:42:20.458169 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 30 01:42:20.458169 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 30 01:42:20.458169 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 30 01:42:20.459035 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 30 01:42:20.459035 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 30 01:42:20.459035 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 30 01:42:20.459035 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 30 01:42:20.459035 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 01:42:20.459035 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 01:42:20.459035 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 01:42:20.459035 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 01:42:20.459035 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 01:42:20.459035 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 01:42:20.459035 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 01:42:20.459035 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application with Lock( Aug 30 01:42:20.459035 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 01:42:20.459035 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._enter() Aug 30 01:42:20.459035 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 01:42:20.459035 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 01:42:20.459812 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 01:42:20.459812 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.creator() Aug 30 01:42:20.459812 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 01:42:20.459812 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application created_value = creator( Aug 30 01:42:20.459812 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 30 01:42:20.459812 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 30 01:42:20.459812 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 30 01:42:20.459812 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 30 01:42:20.459812 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 30 01:42:20.459812 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 30 01:42:20.459812 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 30 01:42:20.459812 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application Aug 30 01:42:20.462101 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 50/104] 10.222.0.27 () {56 vars in 1096 bytes} [Tue Aug 30 01:42:20 2022] GET /identity/v3/roles/admin => generated 83 bytes in 198 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 01:42:20.468301 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ea833c2-89ac-4353-a948-05a148a0cccf None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:42:20.597698 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ea833c2-89ac-4353-a948-05a148a0cccf None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:42:20.643565 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0525bca7-4667-4308-b95f-49a73b2337e0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:42:20.644593 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0525bca7-4667-4308-b95f-49a73b2337e0 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:20.645072 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0525bca7-4667-4308-b95f-49a73b2337e0 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:20.645604 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0525bca7-4667-4308-b95f-49a73b2337e0 None admin] PATH_INFO: `/v3/roles` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:20.646479 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0525bca7-4667-4308-b95f-49a73b2337e0 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:42:20.646979 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0525bca7-4667-4308-b95f-49a73b2337e0 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:42:20.648564 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0525bca7-4667-4308-b95f-49a73b2337e0 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:42:20.661344 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 55/105] 10.222.0.27 () {56 vars in 1099 bytes} [Tue Aug 30 01:42:20 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 194 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:42:20.667164 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55380345-ee11-406f-94e1-ab0059afb019 None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:42:20.674209 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0525bca7-4667-4308-b95f-49a73b2337e0 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:42:20.824198 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:42:20.874554 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0525bca7-4667-4308-b95f-49a73b2337e0 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:42:20.915206 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7816e637-e589-4da9-a370-d67ef9492a4a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:42:20.916658 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7816e637-e589-4da9-a370-d67ef9492a4a None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:20.917423 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7816e637-e589-4da9-a370-d67ef9492a4a None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:20.917861 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7816e637-e589-4da9-a370-d67ef9492a4a None admin] PATH_INFO: `/v3/users/ac83d5ed111c4f4e8b3c3226bd3bbaba` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:20.919188 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7816e637-e589-4da9-a370-d67ef9492a4a None admin] RBAC: Authorizing `identity:get_user(user_id=ac83d5ed111c4f4e8b3c3226bd3bbaba)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:42:20.920867 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7816e637-e589-4da9-a370-d67ef9492a4a None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:42:20.922662 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 56/106] 10.222.0.27 () {56 vars in 1177 bytes} [Tue Aug 30 01:42:20 2022] GET /identity/v3/users/ac83d5ed111c4f4e8b3c3226bd3bbaba => generated 247 bytes in 249 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 30 01:42:20.923657 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7816e637-e589-4da9-a370-d67ef9492a4a None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:42:20.940483 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55380345-ee11-406f-94e1-ab0059afb019 None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:42:20.953831 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:42:20.980916 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed734aba-16f5-4c8b-9f83-99fe506bb4e3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:42:20.981954 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed734aba-16f5-4c8b-9f83-99fe506bb4e3 None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:20.982610 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed734aba-16f5-4c8b-9f83-99fe506bb4e3 None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:20.983085 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed734aba-16f5-4c8b-9f83-99fe506bb4e3 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:20.995344 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed734aba-16f5-4c8b-9f83-99fe506bb4e3 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 30 01:42:21.001276 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application [None req-ed734aba-16f5-4c8b-9f83-99fe506bb4e3 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Aug 30 01:42:21.001276 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 01:42:21.001276 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 01:42:21.001276 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 01:42:21.001276 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 01:42:21.001276 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 01:42:21.001276 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 01:42:21.001276 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 01:42:21.001276 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 01:42:21.001276 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 01:42:21.001276 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 01:42:21.001276 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 01:42:21.001276 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 30 01:42:21.001276 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 30 01:42:21.001276 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 30 01:42:21.001276 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 30 01:42:21.002198 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 30 01:42:21.002198 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 30 01:42:21.002198 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 30 01:42:21.002198 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 30 01:42:21.002198 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 01:42:21.002198 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 01:42:21.002198 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 01:42:21.002198 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 01:42:21.002198 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 01:42:21.002198 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 01:42:21.002198 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 01:42:21.002198 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application with Lock( Aug 30 01:42:21.002198 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 01:42:21.002198 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._enter() Aug 30 01:42:21.002198 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 01:42:21.002198 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 01:42:21.002964 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 01:42:21.002964 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.creator() Aug 30 01:42:21.002964 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 01:42:21.002964 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application created_value = creator( Aug 30 01:42:21.002964 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 30 01:42:21.002964 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 30 01:42:21.002964 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 30 01:42:21.002964 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 30 01:42:21.002964 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 30 01:42:21.002964 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 30 01:42:21.002964 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Aug 30 01:42:21.002964 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application Aug 30 01:42:21.005440 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 51/107] 10.222.0.27 () {56 vars in 1099 bytes} [Tue Aug 30 01:42:20 2022] GET /identity/v3/roles/reader => generated 84 bytes in 340 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 01:42:21.007921 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e77a2434-a7fe-4632-8bf4-ed16fa1b8af2 None None] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:21.008422 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e77a2434-a7fe-4632-8bf4-ed16fa1b8af2 None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:21.008977 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e77a2434-a7fe-4632-8bf4-ed16fa1b8af2 None None] PATH_INFO: `/` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:21.011774 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 52/108] 10.222.0.27 () {54 vars in 927 bytes} [Tue Aug 30 01:42:21 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 01:42:21.012763 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e77a2434-a7fe-4632-8bf4-ed16fa1b8af2 None None] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:42:21.034999 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7816e637-e589-4da9-a370-d67ef9492a4a None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:42:21.042230 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:42:21.070513 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a1dff59-c294-45d4-8a90-624dff076bce None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:42:21.071102 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a1dff59-c294-45d4-8a90-624dff076bce None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:21.071387 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a1dff59-c294-45d4-8a90-624dff076bce None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:21.072589 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a1dff59-c294-45d4-8a90-624dff076bce None admin] PATH_INFO: `/v3/domains/default` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:21.073930 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a1dff59-c294-45d4-8a90-624dff076bce None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:42:21.075529 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a1dff59-c294-45d4-8a90-624dff076bce None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:42:21.077118 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 57/109] 10.222.0.27 () {56 vars in 1108 bytes} [Tue Aug 30 01:42:20 2022] GET /identity/v3/domains/default => generated 199 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:42:21.081782 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a1dff59-c294-45d4-8a90-624dff076bce None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:42:21.139301 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e77a2434-a7fe-4632-8bf4-ed16fa1b8af2 None None] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:42:21.175495 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bff852c5-63e7-481f-908a-ad31ad8f1b0b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:42:21.176124 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bff852c5-63e7-481f-908a-ad31ad8f1b0b None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:21.176405 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bff852c5-63e7-481f-908a-ad31ad8f1b0b None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:21.176675 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bff852c5-63e7-481f-908a-ad31ad8f1b0b None admin] PATH_INFO: `/v3/projects/90818f998d5f4520971eacf6ccc830ce` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:21.177810 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bff852c5-63e7-481f-908a-ad31ad8f1b0b None admin] RBAC: Authorizing `identity:get_project(project_id=90818f998d5f4520971eacf6ccc830ce)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:42:21.179242 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bff852c5-63e7-481f-908a-ad31ad8f1b0b None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:42:21.180864 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 53/110] 10.222.0.27 () {56 vars in 1186 bytes} [Tue Aug 30 01:42:21 2022] GET /identity/v3/projects/90818f998d5f4520971eacf6ccc830ce => generated 298 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:42:21.182574 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83aac6ea-b15d-4fb6-bdc8-28da74307a59 None None] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:21.182852 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83aac6ea-b15d-4fb6-bdc8-28da74307a59 None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:21.183119 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83aac6ea-b15d-4fb6-bdc8-28da74307a59 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:21.194737 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a1dff59-c294-45d4-8a90-624dff076bce None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:42:21.200506 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.auth.core [None req-83aac6ea-b15d-4fb6-bdc8-28da74307a59 None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81578) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 01:42:21.201704 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-83aac6ea-b15d-4fb6-bdc8-28da74307a59 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:42:21.228963 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:42:21.229222 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b88449bf-1ceb-41f1-b13f-7d7178a8828f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:42:21.229610 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b88449bf-1ceb-41f1-b13f-7d7178a8828f None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:21.229900 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b88449bf-1ceb-41f1-b13f-7d7178a8828f None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:21.230173 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b88449bf-1ceb-41f1-b13f-7d7178a8828f None admin] PATH_INFO: `/v3/roles` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:21.230848 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b88449bf-1ceb-41f1-b13f-7d7178a8828f None admin] RBAC: Adding query filter params (name=reader) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:42:21.231154 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b88449bf-1ceb-41f1-b13f-7d7178a8828f None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:42:21.232536 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b88449bf-1ceb-41f1-b13f-7d7178a8828f None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:42:21.241065 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 58/111] 10.222.0.27 () {56 vars in 1101 bytes} [Tue Aug 30 01:42:21 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:42:21.241065 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b88449bf-1ceb-41f1-b13f-7d7178a8828f None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:42:21.268663 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 54/112] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Aug 30 01:42:21 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 30 01:42:21.279533 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbd7961d-0617-40a5-8a5e-1eb02f062308 None None] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:21.279533 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbd7961d-0617-40a5-8a5e-1eb02f062308 None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:21.279533 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbd7961d-0617-40a5-8a5e-1eb02f062308 None None] PATH_INFO: `/` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:21.279533 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 55/113] 10.222.0.27 () {54 vars in 927 bytes} [Tue Aug 30 01:42:21 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 01:42:21.279533 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01e778e8-120f-4a27-9d81-ef1c32591f15 None None] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:21.279533 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01e778e8-120f-4a27-9d81-ef1c32591f15 None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:21.279970 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01e778e8-120f-4a27-9d81-ef1c32591f15 None None] PATH_INFO: `/` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:21.279970 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 56/114] 10.222.0.27 () {54 vars in 927 bytes} [Tue Aug 30 01:42:21 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 01:42:21.279970 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01e778e8-120f-4a27-9d81-ef1c32591f15 None None] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:42:21.290830 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:42:21.362554 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b88449bf-1ceb-41f1-b13f-7d7178a8828f None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:42:21.385242 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01e778e8-120f-4a27-9d81-ef1c32591f15 None None] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:42:21.400804 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ec359fb-fd53-4ffb-8abd-0b245e08031c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:42:21.401254 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ec359fb-fd53-4ffb-8abd-0b245e08031c None admin] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:21.401566 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ec359fb-fd53-4ffb-8abd-0b245e08031c None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:21.401845 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ec359fb-fd53-4ffb-8abd-0b245e08031c None admin] PATH_INFO: `/v3/projects` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:21.402502 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ec359fb-fd53-4ffb-8abd-0b245e08031c None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:42:21.404195 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ec359fb-fd53-4ffb-8abd-0b245e08031c None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:42:21.422477 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d35a0d5b-1ae1-4378-9b7f-4059b1e1fa69 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:42:21.423143 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d35a0d5b-1ae1-4378-9b7f-4059b1e1fa69 None admin] REQUEST_METHOD: `PUT` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:21.423429 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d35a0d5b-1ae1-4378-9b7f-4059b1e1fa69 None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:21.423707 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d35a0d5b-1ae1-4378-9b7f-4059b1e1fa69 None admin] PATH_INFO: `/v3/projects/90818f998d5f4520971eacf6ccc830ce/users/ac83d5ed111c4f4e8b3c3226bd3bbaba/roles/ead5fac86a0747ea8823a82efcea8ed7` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:21.425641 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d35a0d5b-1ae1-4378-9b7f-4059b1e1fa69 None admin] RBAC: Authorizing `identity:create_grant(project_id=90818f998d5f4520971eacf6ccc830ce, user_id=ac83d5ed111c4f4e8b3c3226bd3bbaba, role_id=ead5fac86a0747ea8823a82efcea8ed7)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:42:21.427084 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d35a0d5b-1ae1-4378-9b7f-4059b1e1fa69 None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:42:21.484852 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 59/115] 10.222.0.27 () {60 vars in 1141 bytes} [Tue Aug 30 01:42:21 2022] POST /identity/v3/projects => generated 302 bytes in 243 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 01:42:21.485728 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ec359fb-fd53-4ffb-8abd-0b245e08031c None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:42:21.514612 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 57/116] 10.222.0.27 () {58 vars in 1439 bytes} [Tue Aug 30 01:42:21 2022] PUT /identity/v3/projects/90818f998d5f4520971eacf6ccc830ce/users/ac83d5ed111c4f4e8b3c3226bd3bbaba/roles/ead5fac86a0747ea8823a82efcea8ed7 => generated 0 bytes in 239 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 01:42:21.516206 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0adaec1c-3440-4443-b290-cc549db0a660 None None] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:21.516642 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0adaec1c-3440-4443-b290-cc549db0a660 None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:21.516979 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0adaec1c-3440-4443-b290-cc549db0a660 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:21.539092 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.auth.core [None req-0adaec1c-3440-4443-b290-cc549db0a660 None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81578) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 01:42:21.541367 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-0adaec1c-3440-4443-b290-cc549db0a660 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:42:21.563999 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:42:21.630385 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:42:21.638081 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 58/117] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Aug 30 01:42:21 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 123 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 30 01:42:21.639941 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33cf2bef-969a-4ba6-a09a-8f5a8e3c6128 None None] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:21.640474 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33cf2bef-969a-4ba6-a09a-8f5a8e3c6128 None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:21.640959 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33cf2bef-969a-4ba6-a09a-8f5a8e3c6128 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:21.646475 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ec359fb-fd53-4ffb-8abd-0b245e08031c None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:42:21.663334 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.auth.core [None req-33cf2bef-969a-4ba6-a09a-8f5a8e3c6128 None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81578) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 01:42:21.665018 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-33cf2bef-969a-4ba6-a09a-8f5a8e3c6128 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:42:21.693023 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d9087c8-7875-451c-8571-ef78d08175f7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:42:21.693710 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d9087c8-7875-451c-8571-ef78d08175f7 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:21.693996 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d9087c8-7875-451c-8571-ef78d08175f7 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:21.694333 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d9087c8-7875-451c-8571-ef78d08175f7 None admin] PATH_INFO: `/v3/users/c9207557429f4320ba488c1bffb49e5b` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:21.722233 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d9087c8-7875-451c-8571-ef78d08175f7 None admin] RBAC: Authorizing `identity:get_user(user_id=c9207557429f4320ba488c1bffb49e5b)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:42:21.723739 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d9087c8-7875-451c-8571-ef78d08175f7 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:42:21.725734 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 60/118] 10.222.0.27 () {56 vars in 1177 bytes} [Tue Aug 30 01:42:21 2022] GET /identity/v3/users/c9207557429f4320ba488c1bffb49e5b => generated 289 bytes in 240 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:42:21.726575 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d9087c8-7875-451c-8571-ef78d08175f7 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:42:21.778671 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 59/119] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Aug 30 01:42:21 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 140 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 30 01:42:21.780262 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33cf2bef-969a-4ba6-a09a-8f5a8e3c6128 None None] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:42:21.888283 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:42:21.914348 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33cf2bef-969a-4ba6-a09a-8f5a8e3c6128 None None] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:42:21.952402 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d9087c8-7875-451c-8571-ef78d08175f7 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:42:21.963125 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5861ad32-3890-42f4-a36c-85fd8d077a96 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:42:21.965148 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5861ad32-3890-42f4-a36c-85fd8d077a96 None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:21.965682 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5861ad32-3890-42f4-a36c-85fd8d077a96 None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:21.966164 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5861ad32-3890-42f4-a36c-85fd8d077a96 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:22.039370 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9cec0f0-555c-42a0-843b-878a1586929b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:42:22.040022 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9cec0f0-555c-42a0-843b-878a1586929b None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:22.040303 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9cec0f0-555c-42a0-843b-878a1586929b None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:22.041288 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9cec0f0-555c-42a0-843b-878a1586929b None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:22.052862 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9cec0f0-555c-42a0-843b-878a1586929b None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 30 01:42:22.055666 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5861ad32-3890-42f4-a36c-85fd8d077a96 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:42:22.057670 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5861ad32-3890-42f4-a36c-85fd8d077a96 None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:42:22.059090 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application [None req-d9cec0f0-555c-42a0-843b-878a1586929b None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Aug 30 01:42:22.059090 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 01:42:22.059090 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 01:42:22.059090 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 01:42:22.059090 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 01:42:22.059090 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 01:42:22.059090 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 01:42:22.059090 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 01:42:22.059090 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 01:42:22.059090 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 01:42:22.059090 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 01:42:22.059090 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 01:42:22.059090 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 30 01:42:22.059090 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 30 01:42:22.059090 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 30 01:42:22.059090 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 30 01:42:22.059930 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 30 01:42:22.059930 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 30 01:42:22.059930 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 30 01:42:22.059930 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 30 01:42:22.059930 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 01:42:22.059930 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 01:42:22.059930 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 01:42:22.059930 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 01:42:22.059930 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 01:42:22.059930 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 01:42:22.059930 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 01:42:22.059930 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application with Lock( Aug 30 01:42:22.059930 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 01:42:22.059930 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._enter() Aug 30 01:42:22.059930 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 01:42:22.059930 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 01:42:22.060695 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 01:42:22.060695 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.creator() Aug 30 01:42:22.060695 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 01:42:22.060695 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application created_value = creator( Aug 30 01:42:22.060695 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 30 01:42:22.060695 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 30 01:42:22.060695 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 30 01:42:22.060695 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 30 01:42:22.060695 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 30 01:42:22.060695 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 30 01:42:22.060695 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Aug 30 01:42:22.060695 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Aug 30 01:42:22.062050 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 61/120] 10.222.0.27 () {56 vars in 1114 bytes} [Tue Aug 30 01:42:21 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 335 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 01:42:22.062574 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9cec0f0-555c-42a0-843b-878a1586929b None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:42:22.082694 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:42:22.201923 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9cec0f0-555c-42a0-843b-878a1586929b None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:42:22.239694 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:42:22.263123 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88d8c39f-8a59-4b45-94db-ee111f30e2fc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:42:22.263773 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88d8c39f-8a59-4b45-94db-ee111f30e2fc None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:22.264052 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88d8c39f-8a59-4b45-94db-ee111f30e2fc None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:22.264321 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88d8c39f-8a59-4b45-94db-ee111f30e2fc None admin] PATH_INFO: `/v3/roles/member` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:22.267225 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 60/121] 10.222.0.27 () {58 vars in 1282 bytes} [Tue Aug 30 01:42:21 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 487 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 30 01:42:22.272061 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5861ad32-3890-42f4-a36c-85fd8d077a96 None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:42:22.281003 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88d8c39f-8a59-4b45-94db-ee111f30e2fc None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 30 01:42:22.287265 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application [None req-88d8c39f-8a59-4b45-94db-ee111f30e2fc None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Aug 30 01:42:22.287265 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 01:42:22.287265 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 01:42:22.287265 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 01:42:22.287265 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 01:42:22.287265 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 01:42:22.287265 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 01:42:22.287265 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 01:42:22.287265 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 01:42:22.287265 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 01:42:22.287265 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 01:42:22.287265 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 01:42:22.287265 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 30 01:42:22.287265 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 30 01:42:22.287265 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 30 01:42:22.287265 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 30 01:42:22.288135 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 30 01:42:22.288135 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 30 01:42:22.288135 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 30 01:42:22.288135 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 30 01:42:22.288135 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 01:42:22.288135 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 01:42:22.288135 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 01:42:22.288135 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 01:42:22.288135 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 01:42:22.288135 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 01:42:22.288135 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 01:42:22.288135 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application with Lock( Aug 30 01:42:22.288135 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 01:42:22.288135 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._enter() Aug 30 01:42:22.288135 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 01:42:22.288135 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 01:42:22.288910 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 01:42:22.288910 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.creator() Aug 30 01:42:22.288910 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 01:42:22.288910 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application created_value = creator( Aug 30 01:42:22.288910 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 30 01:42:22.288910 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 30 01:42:22.288910 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 30 01:42:22.288910 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 30 01:42:22.288910 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 30 01:42:22.288910 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 30 01:42:22.288910 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Aug 30 01:42:22.288910 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Aug 30 01:42:22.291540 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 62/122] 10.222.0.27 () {56 vars in 1099 bytes} [Tue Aug 30 01:42:22 2022] GET /identity/v3/roles/member => generated 84 bytes in 230 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 01:42:22.297940 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88d8c39f-8a59-4b45-94db-ee111f30e2fc None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:42:22.380711 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:42:22.421428 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5861ad32-3890-42f4-a36c-85fd8d077a96 None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:42:22.436255 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88d8c39f-8a59-4b45-94db-ee111f30e2fc None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:42:22.465206 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de29b27e-f2f1-4372-b42e-c24d9e9a7e84 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:42:22.465779 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de29b27e-f2f1-4372-b42e-c24d9e9a7e84 None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:22.466058 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de29b27e-f2f1-4372-b42e-c24d9e9a7e84 None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:22.466442 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de29b27e-f2f1-4372-b42e-c24d9e9a7e84 None admin] PATH_INFO: `/v3/roles` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:22.467177 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de29b27e-f2f1-4372-b42e-c24d9e9a7e84 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81578) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:42:22.467479 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de29b27e-f2f1-4372-b42e-c24d9e9a7e84 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:42:22.469104 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de29b27e-f2f1-4372-b42e-c24d9e9a7e84 None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:42:22.481128 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 61/123] 10.222.0.27 () {56 vars in 1111 bytes} [Tue Aug 30 01:42:22 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 213 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:42:22.486567 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de29b27e-f2f1-4372-b42e-c24d9e9a7e84 None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:42:22.488324 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59e6c059-e496-4725-8eb8-bf14d7a45623 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:42:22.489248 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59e6c059-e496-4725-8eb8-bf14d7a45623 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:22.489761 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59e6c059-e496-4725-8eb8-bf14d7a45623 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:22.490558 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59e6c059-e496-4725-8eb8-bf14d7a45623 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:22.507715 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59e6c059-e496-4725-8eb8-bf14d7a45623 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 30 01:42:22.515313 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application [None req-59e6c059-e496-4725-8eb8-bf14d7a45623 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Aug 30 01:42:22.515313 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 01:42:22.515313 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 01:42:22.515313 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 01:42:22.515313 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 01:42:22.515313 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 01:42:22.515313 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 01:42:22.515313 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 01:42:22.515313 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 01:42:22.515313 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 01:42:22.515313 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 01:42:22.515313 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 01:42:22.515313 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 30 01:42:22.515313 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 30 01:42:22.515313 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 30 01:42:22.515313 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 30 01:42:22.516269 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 30 01:42:22.516269 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 30 01:42:22.516269 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 30 01:42:22.516269 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 30 01:42:22.516269 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 01:42:22.516269 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 01:42:22.516269 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 01:42:22.516269 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 01:42:22.516269 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 01:42:22.516269 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 01:42:22.516269 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 01:42:22.516269 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application with Lock( Aug 30 01:42:22.516269 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 01:42:22.516269 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._enter() Aug 30 01:42:22.516269 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 01:42:22.516269 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 01:42:22.517043 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 01:42:22.517043 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.creator() Aug 30 01:42:22.517043 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 01:42:22.517043 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application created_value = creator( Aug 30 01:42:22.517043 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 30 01:42:22.517043 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 30 01:42:22.517043 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 30 01:42:22.517043 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 30 01:42:22.517043 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 30 01:42:22.517043 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 30 01:42:22.517043 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Aug 30 01:42:22.517043 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Aug 30 01:42:22.517043 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 63/124] 10.222.0.27 () {56 vars in 1099 bytes} [Tue Aug 30 01:42:22 2022] GET /identity/v3/roles/member => generated 84 bytes in 225 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 01:42:22.522934 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59e6c059-e496-4725-8eb8-bf14d7a45623 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:42:22.553784 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:42:22.630743 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de29b27e-f2f1-4372-b42e-c24d9e9a7e84 None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:42:22.674145 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59e6c059-e496-4725-8eb8-bf14d7a45623 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:42:22.681690 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cbe6344-fed1-4dbc-b46b-3aa070794dc7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:42:22.682764 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cbe6344-fed1-4dbc-b46b-3aa070794dc7 None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:22.683773 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cbe6344-fed1-4dbc-b46b-3aa070794dc7 None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:22.684230 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cbe6344-fed1-4dbc-b46b-3aa070794dc7 None admin] PATH_INFO: `/v3/projects/90818f998d5f4520971eacf6ccc830ce` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:22.686012 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0cbe6344-fed1-4dbc-b46b-3aa070794dc7 None admin] RBAC: Authorizing `identity:get_project(project_id=90818f998d5f4520971eacf6ccc830ce)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:42:22.688439 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0cbe6344-fed1-4dbc-b46b-3aa070794dc7 None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:42:22.690543 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 62/125] 10.222.0.27 () {56 vars in 1186 bytes} [Tue Aug 30 01:42:22 2022] GET /identity/v3/projects/90818f998d5f4520971eacf6ccc830ce => generated 298 bytes in 209 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:42:22.691903 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cbe6344-fed1-4dbc-b46b-3aa070794dc7 None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:42:22.711654 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:42:22.738225 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b942b3e7-0e66-4123-b5e1-06bb7b3462a0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:42:22.740385 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b942b3e7-0e66-4123-b5e1-06bb7b3462a0 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:22.741224 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b942b3e7-0e66-4123-b5e1-06bb7b3462a0 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:22.742268 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b942b3e7-0e66-4123-b5e1-06bb7b3462a0 None admin] PATH_INFO: `/v3/roles` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:22.744166 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b942b3e7-0e66-4123-b5e1-06bb7b3462a0 None admin] RBAC: Adding query filter params (name=member) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:42:22.745232 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b942b3e7-0e66-4123-b5e1-06bb7b3462a0 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:42:22.747811 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b942b3e7-0e66-4123-b5e1-06bb7b3462a0 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:42:22.760127 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 64/126] 10.222.0.27 () {56 vars in 1101 bytes} [Tue Aug 30 01:42:22 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 243 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:42:22.765595 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b942b3e7-0e66-4123-b5e1-06bb7b3462a0 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:42:22.844855 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cbe6344-fed1-4dbc-b46b-3aa070794dc7 None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:42:22.864404 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:42:22.885376 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b942b3e7-0e66-4123-b5e1-06bb7b3462a0 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:42:22.892416 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efe03f61-32b7-4b3d-910e-2988ef58ed45 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:42:22.893788 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efe03f61-32b7-4b3d-910e-2988ef58ed45 None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:22.894272 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efe03f61-32b7-4b3d-910e-2988ef58ed45 None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:22.894732 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efe03f61-32b7-4b3d-910e-2988ef58ed45 None admin] PATH_INFO: `/v3/users/fac663577ae547588ada9d16b2fd6f5f` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:22.896459 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efe03f61-32b7-4b3d-910e-2988ef58ed45 None admin] RBAC: Authorizing `identity:get_user(user_id=fac663577ae547588ada9d16b2fd6f5f)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:42:22.898780 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efe03f61-32b7-4b3d-910e-2988ef58ed45 None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:42:22.902830 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 63/127] 10.222.0.27 () {56 vars in 1177 bytes} [Tue Aug 30 01:42:22 2022] GET /identity/v3/users/fac663577ae547588ada9d16b2fd6f5f => generated 275 bytes in 211 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:42:22.904248 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efe03f61-32b7-4b3d-910e-2988ef58ed45 None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:42:22.932204 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1617c8ec-b6c4-4448-966e-5db7ac47e138 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:42:22.932936 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1617c8ec-b6c4-4448-966e-5db7ac47e138 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:22.933286 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1617c8ec-b6c4-4448-966e-5db7ac47e138 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:22.933660 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1617c8ec-b6c4-4448-966e-5db7ac47e138 None admin] PATH_INFO: `/v3/roles` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:22.934462 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1617c8ec-b6c4-4448-966e-5db7ac47e138 None admin] RBAC: Adding query filter params (name=member) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:42:22.934841 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1617c8ec-b6c4-4448-966e-5db7ac47e138 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:42:22.936520 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1617c8ec-b6c4-4448-966e-5db7ac47e138 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:42:22.951791 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 65/128] 10.222.0.27 () {56 vars in 1101 bytes} [Tue Aug 30 01:42:22 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:42:22.951791 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1617c8ec-b6c4-4448-966e-5db7ac47e138 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:42:23.026216 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:42:23.060233 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efe03f61-32b7-4b3d-910e-2988ef58ed45 None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:42:23.099180 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1617c8ec-b6c4-4448-966e-5db7ac47e138 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:42:23.103751 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee5f654c-9395-4a59-ab1d-ff16b2429e7f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:42:23.104490 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee5f654c-9395-4a59-ab1d-ff16b2429e7f None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:23.104909 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee5f654c-9395-4a59-ab1d-ff16b2429e7f None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:23.105306 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee5f654c-9395-4a59-ab1d-ff16b2429e7f None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:23.106216 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee5f654c-9395-4a59-ab1d-ff16b2429e7f None admin] RBAC: Adding query filter params (role.id=ffde3a55b5dd43149c18c72b7f2fe6ae, scope.project.id=90818f998d5f4520971eacf6ccc830ce, user.id=c9207557429f4320ba488c1bffb49e5b) {{(pid=81578) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:42:23.106688 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee5f654c-9395-4a59-ab1d-ff16b2429e7f None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:42:23.108361 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee5f654c-9395-4a59-ab1d-ff16b2429e7f None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:42:23.115228 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 64/129] 10.222.0.27 () {56 vars in 1374 bytes} [Tue Aug 30 01:42:22 2022] GET /identity/v3/role_assignments?user.id=c9207557429f4320ba488c1bffb49e5b&scope.project.id=90818f998d5f4520971eacf6ccc830ce&role.id=ffde3a55b5dd43149c18c72b7f2fe6ae => generated 259 bytes in 211 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:42:23.120947 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee5f654c-9395-4a59-ab1d-ff16b2429e7f None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:42:23.141933 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdd27eb9-f270-4745-9dd2-a352795c1301 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:42:23.143408 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdd27eb9-f270-4745-9dd2-a352795c1301 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:23.143832 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdd27eb9-f270-4745-9dd2-a352795c1301 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:23.144233 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdd27eb9-f270-4745-9dd2-a352795c1301 None admin] PATH_INFO: `/v3/users/fac663577ae547588ada9d16b2fd6f5f` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:23.145697 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bdd27eb9-f270-4745-9dd2-a352795c1301 None admin] RBAC: Authorizing `identity:get_user(user_id=fac663577ae547588ada9d16b2fd6f5f)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:42:23.147764 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bdd27eb9-f270-4745-9dd2-a352795c1301 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:42:23.150561 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 66/130] 10.222.0.27 () {56 vars in 1177 bytes} [Tue Aug 30 01:42:22 2022] GET /identity/v3/users/fac663577ae547588ada9d16b2fd6f5f => generated 275 bytes in 203 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:42:23.157028 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdd27eb9-f270-4745-9dd2-a352795c1301 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:42:23.225053 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:42:23.283370 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdd27eb9-f270-4745-9dd2-a352795c1301 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:42:23.310582 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:42:23.327602 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a605da7c-dac9-42b6-b1aa-10554c336998 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:42:23.329425 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a605da7c-dac9-42b6-b1aa-10554c336998 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:23.330105 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a605da7c-dac9-42b6-b1aa-10554c336998 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:23.330710 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a605da7c-dac9-42b6-b1aa-10554c336998 None admin] PATH_INFO: `/v3/users/fac663577ae547588ada9d16b2fd6f5f` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:23.332360 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a605da7c-dac9-42b6-b1aa-10554c336998 None admin] RBAC: Authorizing `identity:get_user(user_id=fac663577ae547588ada9d16b2fd6f5f)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:42:23.334697 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a605da7c-dac9-42b6-b1aa-10554c336998 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:42:23.337056 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 67/131] 10.222.0.27 () {56 vars in 1177 bytes} [Tue Aug 30 01:42:23 2022] GET /identity/v3/users/fac663577ae547588ada9d16b2fd6f5f => generated 275 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:42:23.343128 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a605da7c-dac9-42b6-b1aa-10554c336998 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:42:23.361005 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee5f654c-9395-4a59-ab1d-ff16b2429e7f None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:42:23.429285 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac6269b1-f978-4171-8643-f06f9549087c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:42:23.430253 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac6269b1-f978-4171-8643-f06f9549087c None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:23.430579 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac6269b1-f978-4171-8643-f06f9549087c None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:23.430856 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac6269b1-f978-4171-8643-f06f9549087c None admin] PATH_INFO: `/v3/projects/90818f998d5f4520971eacf6ccc830ce` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:23.433039 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac6269b1-f978-4171-8643-f06f9549087c None admin] RBAC: Authorizing `identity:get_project(project_id=90818f998d5f4520971eacf6ccc830ce)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:42:23.437494 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac6269b1-f978-4171-8643-f06f9549087c None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:42:23.438273 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 65/132] 10.222.0.27 () {56 vars in 1186 bytes} [Tue Aug 30 01:42:23 2022] GET /identity/v3/projects/90818f998d5f4520971eacf6ccc830ce => generated 298 bytes in 323 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:42:23.439872 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac6269b1-f978-4171-8643-f06f9549087c None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:42:23.570109 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a605da7c-dac9-42b6-b1aa-10554c336998 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:42:23.635163 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac6269b1-f978-4171-8643-f06f9549087c None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:42:23.639963 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85e8bb7c-88f7-4a34-84d2-f808597ce5a9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:42:23.640616 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85e8bb7c-88f7-4a34-84d2-f808597ce5a9 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:23.640890 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85e8bb7c-88f7-4a34-84d2-f808597ce5a9 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:23.641173 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85e8bb7c-88f7-4a34-84d2-f808597ce5a9 None admin] PATH_INFO: `/v3/projects/90818f998d5f4520971eacf6ccc830ce` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:23.646748 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85e8bb7c-88f7-4a34-84d2-f808597ce5a9 None admin] RBAC: Authorizing `identity:get_project(project_id=90818f998d5f4520971eacf6ccc830ce)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:42:23.648233 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85e8bb7c-88f7-4a34-84d2-f808597ce5a9 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:42:23.650314 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 68/133] 10.222.0.27 () {56 vars in 1186 bytes} [Tue Aug 30 01:42:23 2022] GET /identity/v3/projects/90818f998d5f4520971eacf6ccc830ce => generated 298 bytes in 312 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:42:23.651090 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85e8bb7c-88f7-4a34-84d2-f808597ce5a9 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:42:23.667490 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:42:23.709877 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0ca2567-20e8-41dd-a5a9-3344e0e68bf4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:42:23.710526 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0ca2567-20e8-41dd-a5a9-3344e0e68bf4 None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:23.710808 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0ca2567-20e8-41dd-a5a9-3344e0e68bf4 None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:23.711079 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0ca2567-20e8-41dd-a5a9-3344e0e68bf4 None admin] PATH_INFO: `/v3/projects/2b886b3234b7472ca4968ccca426a61d` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:23.712267 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0ca2567-20e8-41dd-a5a9-3344e0e68bf4 None admin] RBAC: Authorizing `identity:get_project(project_id=2b886b3234b7472ca4968ccca426a61d)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:42:23.713746 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0ca2567-20e8-41dd-a5a9-3344e0e68bf4 None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:42:23.715825 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 66/134] 10.222.0.27 () {56 vars in 1186 bytes} [Tue Aug 30 01:42:23 2022] GET /identity/v3/projects/2b886b3234b7472ca4968ccca426a61d => generated 312 bytes in 277 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:42:23.716312 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0ca2567-20e8-41dd-a5a9-3344e0e68bf4 None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:42:23.743023 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:42:23.797434 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85e8bb7c-88f7-4a34-84d2-f808597ce5a9 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:42:23.840377 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-443cad1b-1e03-4964-9056-df58ea21a074 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:42:23.841142 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-443cad1b-1e03-4964-9056-df58ea21a074 None admin] REQUEST_METHOD: `PUT` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:23.841563 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-443cad1b-1e03-4964-9056-df58ea21a074 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:23.842041 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-443cad1b-1e03-4964-9056-df58ea21a074 None admin] PATH_INFO: `/v3/projects/90818f998d5f4520971eacf6ccc830ce/users/fac663577ae547588ada9d16b2fd6f5f/roles/4621068972ba4f79bad104f956db04aa` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:23.844609 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-443cad1b-1e03-4964-9056-df58ea21a074 None admin] RBAC: Authorizing `identity:create_grant(project_id=90818f998d5f4520971eacf6ccc830ce, user_id=fac663577ae547588ada9d16b2fd6f5f, role_id=4621068972ba4f79bad104f956db04aa)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:42:23.846472 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-443cad1b-1e03-4964-9056-df58ea21a074 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:42:23.853257 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0ca2567-20e8-41dd-a5a9-3344e0e68bf4 None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:42:23.891104 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca1027c6-5990-40f9-8dff-4a67fecde64e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:42:23.891768 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca1027c6-5990-40f9-8dff-4a67fecde64e None admin] REQUEST_METHOD: `PUT` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:23.892049 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca1027c6-5990-40f9-8dff-4a67fecde64e None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:23.892321 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca1027c6-5990-40f9-8dff-4a67fecde64e None admin] PATH_INFO: `/v3/projects/90818f998d5f4520971eacf6ccc830ce/users/fac663577ae547588ada9d16b2fd6f5f/roles/1fb81afcb9f245f3b989add94598a063` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:23.894630 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca1027c6-5990-40f9-8dff-4a67fecde64e None admin] RBAC: Authorizing `identity:create_grant(project_id=90818f998d5f4520971eacf6ccc830ce, user_id=fac663577ae547588ada9d16b2fd6f5f, role_id=1fb81afcb9f245f3b989add94598a063)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:42:23.895896 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca1027c6-5990-40f9-8dff-4a67fecde64e None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:42:23.941497 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 69/135] 10.222.0.27 () {58 vars in 1439 bytes} [Tue Aug 30 01:42:23 2022] PUT /identity/v3/projects/90818f998d5f4520971eacf6ccc830ce/users/fac663577ae547588ada9d16b2fd6f5f/roles/4621068972ba4f79bad104f956db04aa => generated 0 bytes in 284 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 01:42:23.941497 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9417a2e-40fa-44df-af2f-1a2fe653fa2e None None] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:23.941497 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9417a2e-40fa-44df-af2f-1a2fe653fa2e None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:23.941497 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9417a2e-40fa-44df-af2f-1a2fe653fa2e None None] PATH_INFO: `/` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:23.941497 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 70/136] 10.222.0.27 () {54 vars in 927 bytes} [Tue Aug 30 01:42:23 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 01:42:23.941497 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9417a2e-40fa-44df-af2f-1a2fe653fa2e None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:42:24.001996 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 67/137] 10.222.0.27 () {58 vars in 1439 bytes} [Tue Aug 30 01:42:23 2022] PUT /identity/v3/projects/90818f998d5f4520971eacf6ccc830ce/users/fac663577ae547588ada9d16b2fd6f5f/roles/1fb81afcb9f245f3b989add94598a063 => generated 0 bytes in 286 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 01:42:24.003099 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b7289d0-d588-4f26-b3a6-d3dc24a9c85e None None] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:24.003449 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b7289d0-d588-4f26-b3a6-d3dc24a9c85e None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:24.003672 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b7289d0-d588-4f26-b3a6-d3dc24a9c85e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:24.045662 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.auth.core [None req-2b7289d0-d588-4f26-b3a6-d3dc24a9c85e None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81578) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 01:42:24.048708 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-2b7289d0-d588-4f26-b3a6-d3dc24a9c85e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:42:24.132868 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9417a2e-40fa-44df-af2f-1a2fe653fa2e None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:42:24.143400 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:42:24.163886 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:42:24.195041 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5615131e-9da0-4e1e-a151-f3e41511fa16 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:42:24.196066 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5615131e-9da0-4e1e-a151-f3e41511fa16 None admin] REQUEST_METHOD: `PUT` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:24.196343 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5615131e-9da0-4e1e-a151-f3e41511fa16 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:24.197055 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5615131e-9da0-4e1e-a151-f3e41511fa16 None admin] PATH_INFO: `/v3/projects/2b886b3234b7472ca4968ccca426a61d/users/fac663577ae547588ada9d16b2fd6f5f/roles/1fb81afcb9f245f3b989add94598a063` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:24.199691 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5615131e-9da0-4e1e-a151-f3e41511fa16 None admin] RBAC: Authorizing `identity:create_grant(project_id=2b886b3234b7472ca4968ccca426a61d, user_id=fac663577ae547588ada9d16b2fd6f5f, role_id=1fb81afcb9f245f3b989add94598a063)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:42:24.201595 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5615131e-9da0-4e1e-a151-f3e41511fa16 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:42:24.208032 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 68/138] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Aug 30 01:42:24 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 206 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 30 01:42:24.209655 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb4c4e4d-d5ab-48ad-a647-0ee95d46b2b7 None None] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:24.209940 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb4c4e4d-d5ab-48ad-a647-0ee95d46b2b7 None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:24.210213 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb4c4e4d-d5ab-48ad-a647-0ee95d46b2b7 None None] PATH_INFO: `/` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:24.211781 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 69/139] 10.222.0.27 () {54 vars in 927 bytes} [Tue Aug 30 01:42:24 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 01:42:24.218092 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3b5c8db-21c2-4e07-ae43-cfd0d31a30ee None None] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:24.218890 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3b5c8db-21c2-4e07-ae43-cfd0d31a30ee None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:24.218890 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3b5c8db-21c2-4e07-ae43-cfd0d31a30ee None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:24.242675 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.auth.core [None req-b3b5c8db-21c2-4e07-ae43-cfd0d31a30ee None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81578) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 01:42:24.244350 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-b3b5c8db-21c2-4e07-ae43-cfd0d31a30ee None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:42:24.311406 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 71/140] 10.222.0.27 () {58 vars in 1439 bytes} [Tue Aug 30 01:42:23 2022] PUT /identity/v3/projects/2b886b3234b7472ca4968ccca426a61d/users/fac663577ae547588ada9d16b2fd6f5f/roles/1fb81afcb9f245f3b989add94598a063 => generated 0 bytes in 372 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 01:42:24.340390 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 70/141] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Aug 30 01:42:24 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 123 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Aug 30 01:42:24.695824 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5615131e-9da0-4e1e-a151-f3e41511fa16 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:42:24.768833 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3b5c8db-21c2-4e07-ae43-cfd0d31a30ee None None] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:42:24.854194 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5615131e-9da0-4e1e-a151-f3e41511fa16 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:42:24.943258 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-210d6915-52cf-46bb-b125-34a740154214 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:42:24.943910 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-210d6915-52cf-46bb-b125-34a740154214 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:24.944202 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-210d6915-52cf-46bb-b125-34a740154214 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:24.944470 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-210d6915-52cf-46bb-b125-34a740154214 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:24.963588 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:42:24.967539 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3b5c8db-21c2-4e07-ae43-cfd0d31a30ee None None] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:42:24.979219 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-210d6915-52cf-46bb-b125-34a740154214 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 30 01:42:24.985285 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application [None req-210d6915-52cf-46bb-b125-34a740154214 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 30 01:42:24.985285 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 01:42:24.985285 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 01:42:24.985285 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 01:42:24.985285 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 01:42:24.985285 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 01:42:24.985285 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 01:42:24.985285 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 01:42:24.985285 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 01:42:24.985285 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 01:42:24.985285 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 01:42:24.985285 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 01:42:24.985285 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 30 01:42:24.985285 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 30 01:42:24.985285 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 30 01:42:24.985285 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 30 01:42:24.986278 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 30 01:42:24.986278 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 30 01:42:24.986278 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 30 01:42:24.986278 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 30 01:42:24.986278 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 01:42:24.986278 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 01:42:24.986278 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 01:42:24.986278 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 01:42:24.986278 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 01:42:24.986278 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 01:42:24.986278 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 01:42:24.986278 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application with Lock( Aug 30 01:42:24.986278 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 01:42:24.986278 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._enter() Aug 30 01:42:24.986278 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 01:42:24.986278 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 01:42:24.987058 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 01:42:24.987058 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.creator() Aug 30 01:42:24.987058 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 01:42:24.987058 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application created_value = creator( Aug 30 01:42:24.987058 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 30 01:42:24.987058 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 30 01:42:24.987058 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 30 01:42:24.987058 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 30 01:42:24.987058 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 30 01:42:24.987058 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 30 01:42:24.987058 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 30 01:42:24.987058 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Aug 30 01:42:24.987058 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 72/142] 10.222.0.27 () {56 vars in 1096 bytes} [Tue Aug 30 01:42:24 2022] GET /identity/v3/roles/admin => generated 83 bytes in 291 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 01:42:24.992536 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-210d6915-52cf-46bb-b125-34a740154214 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:42:25.048585 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1cf3ed3-f9a3-4cf5-9bb7-7c9ba49bcc49 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:42:25.049406 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1cf3ed3-f9a3-4cf5-9bb7-7c9ba49bcc49 None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:25.049717 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1cf3ed3-f9a3-4cf5-9bb7-7c9ba49bcc49 None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:25.050341 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1cf3ed3-f9a3-4cf5-9bb7-7c9ba49bcc49 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:25.054223 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1cf3ed3-f9a3-4cf5-9bb7-7c9ba49bcc49 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:42:25.055700 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1cf3ed3-f9a3-4cf5-9bb7-7c9ba49bcc49 None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:42:25.057935 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 71/143] 10.222.0.27 () {56 vars in 1108 bytes} [Tue Aug 30 01:42:24 2022] GET /identity/v3/domains/default => generated 199 bytes in 290 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 30 01:42:25.064971 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1cf3ed3-f9a3-4cf5-9bb7-7c9ba49bcc49 None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:42:25.065827 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:42:25.192244 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-210d6915-52cf-46bb-b125-34a740154214 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:42:25.252272 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d50b7db-c1ba-4027-8132-c9d4021739e3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:42:25.252792 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d50b7db-c1ba-4027-8132-c9d4021739e3 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:25.253076 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d50b7db-c1ba-4027-8132-c9d4021739e3 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:25.253348 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d50b7db-c1ba-4027-8132-c9d4021739e3 None admin] PATH_INFO: `/v3/roles` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:25.254051 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d50b7db-c1ba-4027-8132-c9d4021739e3 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:42:25.254353 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d50b7db-c1ba-4027-8132-c9d4021739e3 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:42:25.255758 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d50b7db-c1ba-4027-8132-c9d4021739e3 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:42:25.271206 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1cf3ed3-f9a3-4cf5-9bb7-7c9ba49bcc49 None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:42:25.289684 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 73/144] 10.222.0.27 () {56 vars in 1099 bytes} [Tue Aug 30 01:42:24 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 297 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 30 01:42:25.302518 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d50b7db-c1ba-4027-8132-c9d4021739e3 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:42:25.367598 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58e2780e-830f-432b-be10-49da0b34f80d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:42:25.368311 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58e2780e-830f-432b-be10-49da0b34f80d None admin] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:25.368606 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58e2780e-830f-432b-be10-49da0b34f80d None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:25.368884 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58e2780e-830f-432b-be10-49da0b34f80d None admin] PATH_INFO: `/v3/users` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:25.369588 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58e2780e-830f-432b-be10-49da0b34f80d None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:42:25.371047 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58e2780e-830f-432b-be10-49da0b34f80d None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:42:25.485052 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 72/145] 10.222.0.27 () {60 vars in 1132 bytes} [Tue Aug 30 01:42:25 2022] POST /identity/v3/users => generated 283 bytes in 420 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 01:42:25.494982 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:42:25.501193 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d50b7db-c1ba-4027-8132-c9d4021739e3 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:42:25.553000 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab7f958b-25de-433b-93c4-39810e754915 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:42:25.553670 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab7f958b-25de-433b-93c4-39810e754915 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:25.553951 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab7f958b-25de-433b-93c4-39810e754915 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:25.554219 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab7f958b-25de-433b-93c4-39810e754915 None admin] PATH_INFO: `/v3/users/ac83d5ed111c4f4e8b3c3226bd3bbaba` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:25.558267 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab7f958b-25de-433b-93c4-39810e754915 None admin] RBAC: Authorizing `identity:get_user(user_id=ac83d5ed111c4f4e8b3c3226bd3bbaba)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:42:25.559744 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab7f958b-25de-433b-93c4-39810e754915 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:42:25.561760 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 74/146] 10.222.0.27 () {56 vars in 1177 bytes} [Tue Aug 30 01:42:25 2022] GET /identity/v3/users/ac83d5ed111c4f4e8b3c3226bd3bbaba => generated 247 bytes in 260 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:42:25.567869 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58e2780e-830f-432b-be10-49da0b34f80d None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:42:25.626771 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:42:25.812509 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58e2780e-830f-432b-be10-49da0b34f80d None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:42:25.867214 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ce4d1d9-5145-4d1e-94ae-a41d1fe83c1f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:42:25.868209 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ce4d1d9-5145-4d1e-94ae-a41d1fe83c1f None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:25.868734 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ce4d1d9-5145-4d1e-94ae-a41d1fe83c1f None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:25.869208 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ce4d1d9-5145-4d1e-94ae-a41d1fe83c1f None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:25.964909 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ce4d1d9-5145-4d1e-94ae-a41d1fe83c1f None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:42:25.966780 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ce4d1d9-5145-4d1e-94ae-a41d1fe83c1f None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:42:26.006557 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:42:26.105923 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 73/147] 10.222.0.27 () {58 vars in 1282 bytes} [Tue Aug 30 01:42:25 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 538 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 30 01:42:26.112114 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab7f958b-25de-433b-93c4-39810e754915 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:42:26.259933 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab7f958b-25de-433b-93c4-39810e754915 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:42:26.301690 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-429fc760-0a81-401f-95d4-25674b6b4190 None None] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:26.302759 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-429fc760-0a81-401f-95d4-25674b6b4190 None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:26.303235 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-429fc760-0a81-401f-95d4-25674b6b4190 None None] PATH_INFO: `/` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:26.305370 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 74/148] 10.222.0.27 () {54 vars in 927 bytes} [Tue Aug 30 01:42:26 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 01:42:26.311622 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e575bed-b98e-41f6-8d42-9587bb0c9757 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:42:26.312286 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e575bed-b98e-41f6-8d42-9587bb0c9757 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:26.312569 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e575bed-b98e-41f6-8d42-9587bb0c9757 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:26.312841 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e575bed-b98e-41f6-8d42-9587bb0c9757 None admin] PATH_INFO: `/v3/projects/90818f998d5f4520971eacf6ccc830ce` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:26.314066 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e575bed-b98e-41f6-8d42-9587bb0c9757 None admin] RBAC: Authorizing `identity:get_project(project_id=90818f998d5f4520971eacf6ccc830ce)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:42:26.314425 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04603d89-5832-464f-a15d-cffdeffec86b None None] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:26.315207 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04603d89-5832-464f-a15d-cffdeffec86b None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:26.315541 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e575bed-b98e-41f6-8d42-9587bb0c9757 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:42:26.316139 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04603d89-5832-464f-a15d-cffdeffec86b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:26.325087 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 75/149] 10.222.0.27 () {56 vars in 1186 bytes} [Tue Aug 30 01:42:26 2022] GET /identity/v3/projects/90818f998d5f4520971eacf6ccc830ce => generated 298 bytes in 206 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:42:26.326526 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e575bed-b98e-41f6-8d42-9587bb0c9757 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:42:26.353765 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.auth.core [None req-04603d89-5832-464f-a15d-cffdeffec86b None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81578) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 01:42:26.355332 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-04603d89-5832-464f-a15d-cffdeffec86b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:42:26.365961 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:42:26.464353 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e575bed-b98e-41f6-8d42-9587bb0c9757 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:42:26.513324 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c114711-435e-404b-a244-e658a422c8ed None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:42:26.514827 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c114711-435e-404b-a244-e658a422c8ed None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:26.515578 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c114711-435e-404b-a244-e658a422c8ed None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:26.516313 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c114711-435e-404b-a244-e658a422c8ed None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:26.517945 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c114711-435e-404b-a244-e658a422c8ed None admin] RBAC: Adding query filter params (role.id=ead5fac86a0747ea8823a82efcea8ed7, scope.project.id=90818f998d5f4520971eacf6ccc830ce, user.id=ac83d5ed111c4f4e8b3c3226bd3bbaba) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:42:26.518769 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c114711-435e-404b-a244-e658a422c8ed None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:42:26.521865 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c114711-435e-404b-a244-e658a422c8ed None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:42:26.530630 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 76/150] 10.222.0.27 () {56 vars in 1374 bytes} [Tue Aug 30 01:42:26 2022] GET /identity/v3/role_assignments?user.id=ac83d5ed111c4f4e8b3c3226bd3bbaba&scope.project.id=90818f998d5f4520971eacf6ccc830ce&role.id=ead5fac86a0747ea8823a82efcea8ed7 => generated 608 bytes in 205 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 30 01:42:26.548191 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 75/151] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Aug 30 01:42:26 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 236 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Aug 30 01:42:26.612046 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1bfed2e-d3d5-45c8-a098-8ef5cdd6e4a3 None None] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:26.612663 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1bfed2e-d3d5-45c8-a098-8ef5cdd6e4a3 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:26.613210 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1bfed2e-d3d5-45c8-a098-8ef5cdd6e4a3 None None] PATH_INFO: `/` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:26.614819 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 77/152] 10.222.0.27 () {54 vars in 927 bytes} [Tue Aug 30 01:42:26 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 01:42:26.621683 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40ae94f0-80c3-4129-9886-e98498f917ef None None] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:26.621948 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40ae94f0-80c3-4129-9886-e98498f917ef None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:26.622162 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40ae94f0-80c3-4129-9886-e98498f917ef None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:26.644773 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.auth.core [None req-40ae94f0-80c3-4129-9886-e98498f917ef None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81578) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 01:42:26.646032 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-40ae94f0-80c3-4129-9886-e98498f917ef None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:42:26.707484 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:42:26.736246 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 76/153] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Aug 30 01:42:26 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 116 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 30 01:42:27.026498 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1bfed2e-d3d5-45c8-a098-8ef5cdd6e4a3 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:42:27.077687 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a34f9fa-bb2e-4a7c-b48f-81831516743c None None] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:27.078526 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a34f9fa-bb2e-4a7c-b48f-81831516743c None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:27.079992 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a34f9fa-bb2e-4a7c-b48f-81831516743c None None] PATH_INFO: `/` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:27.081917 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 77/154] 10.222.0.27 () {54 vars in 927 bytes} [Tue Aug 30 01:42:27 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 01:42:27.089276 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f15158ce-0049-4fd3-b32f-5ddbd4738fa4 None None] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:27.090158 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f15158ce-0049-4fd3-b32f-5ddbd4738fa4 None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:27.090638 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f15158ce-0049-4fd3-b32f-5ddbd4738fa4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:27.132928 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.auth.core [None req-f15158ce-0049-4fd3-b32f-5ddbd4738fa4 None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81578) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 01:42:27.134281 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-f15158ce-0049-4fd3-b32f-5ddbd4738fa4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:42:27.180624 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:42:27.197724 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1bfed2e-d3d5-45c8-a098-8ef5cdd6e4a3 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:42:27.248967 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cc80ee4-512b-460b-8b89-539327d56c6a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:42:27.249656 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cc80ee4-512b-460b-8b89-539327d56c6a None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:27.249935 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cc80ee4-512b-460b-8b89-539327d56c6a None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:27.250201 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cc80ee4-512b-460b-8b89-539327d56c6a None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:27.264108 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0cc80ee4-512b-460b-8b89-539327d56c6a None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 30 01:42:27.269644 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application [None req-0cc80ee4-512b-460b-8b89-539327d56c6a None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Aug 30 01:42:27.269644 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 01:42:27.269644 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 01:42:27.269644 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 01:42:27.269644 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 01:42:27.269644 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 01:42:27.269644 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 01:42:27.269644 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 01:42:27.269644 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 01:42:27.269644 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 01:42:27.269644 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 01:42:27.269644 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 01:42:27.269644 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 30 01:42:27.269644 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 30 01:42:27.269644 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 30 01:42:27.269644 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 30 01:42:27.270509 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 30 01:42:27.270509 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 30 01:42:27.270509 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 30 01:42:27.270509 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 30 01:42:27.270509 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 01:42:27.270509 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 01:42:27.270509 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 01:42:27.270509 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 01:42:27.270509 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 01:42:27.270509 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 01:42:27.270509 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 01:42:27.270509 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application with Lock( Aug 30 01:42:27.270509 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 01:42:27.270509 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._enter() Aug 30 01:42:27.270509 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 01:42:27.270509 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 01:42:27.271658 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 01:42:27.271658 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.creator() Aug 30 01:42:27.271658 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 01:42:27.271658 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application created_value = creator( Aug 30 01:42:27.271658 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 30 01:42:27.271658 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 30 01:42:27.271658 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 30 01:42:27.271658 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 30 01:42:27.271658 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 30 01:42:27.271658 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 30 01:42:27.271658 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Aug 30 01:42:27.271658 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Aug 30 01:42:27.271658 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 78/155] 10.222.0.27 () {56 vars in 1099 bytes} [Tue Aug 30 01:42:27 2022] GET /identity/v3/roles/reader => generated 84 bytes in 245 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 01:42:27.271658 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cc80ee4-512b-460b-8b89-539327d56c6a None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:42:27.273836 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 78/156] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Aug 30 01:42:27 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 185 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Aug 30 01:42:27.275172 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bfe0b58-7971-411a-a731-750b2c82fd37 None None] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:27.275751 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bfe0b58-7971-411a-a731-750b2c82fd37 None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:27.276024 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bfe0b58-7971-411a-a731-750b2c82fd37 None None] PATH_INFO: `/` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:27.278402 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 79/157] 10.222.0.27 () {54 vars in 927 bytes} [Tue Aug 30 01:42:27 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 01:42:27.283594 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bfe0b58-7971-411a-a731-750b2c82fd37 None None] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:42:27.405822 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cc80ee4-512b-460b-8b89-539327d56c6a None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:42:27.460175 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-130221a5-6a61-439d-a99d-b0191bfe3c6e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:42:27.460828 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-130221a5-6a61-439d-a99d-b0191bfe3c6e None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:27.461105 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-130221a5-6a61-439d-a99d-b0191bfe3c6e None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:27.461372 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-130221a5-6a61-439d-a99d-b0191bfe3c6e None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:27.482867 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bfe0b58-7971-411a-a731-750b2c82fd37 None None] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:42:27.498254 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-130221a5-6a61-439d-a99d-b0191bfe3c6e None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 30 01:42:27.508538 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application [None req-130221a5-6a61-439d-a99d-b0191bfe3c6e None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Aug 30 01:42:27.508538 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 01:42:27.508538 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 01:42:27.508538 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 01:42:27.508538 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 01:42:27.508538 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 01:42:27.508538 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 01:42:27.508538 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 01:42:27.508538 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 01:42:27.508538 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 01:42:27.508538 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 01:42:27.508538 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 01:42:27.508538 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 30 01:42:27.508538 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 30 01:42:27.508538 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 30 01:42:27.508538 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 30 01:42:27.510372 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 30 01:42:27.510372 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 30 01:42:27.510372 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 30 01:42:27.510372 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 30 01:42:27.510372 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 01:42:27.510372 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 01:42:27.510372 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 01:42:27.510372 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 01:42:27.510372 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 01:42:27.510372 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 01:42:27.510372 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 01:42:27.510372 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application with Lock( Aug 30 01:42:27.510372 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 01:42:27.510372 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._enter() Aug 30 01:42:27.510372 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 01:42:27.510372 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 01:42:27.511188 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 01:42:27.511188 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.creator() Aug 30 01:42:27.511188 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 01:42:27.511188 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application created_value = creator( Aug 30 01:42:27.511188 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 30 01:42:27.511188 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 30 01:42:27.511188 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 30 01:42:27.511188 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 30 01:42:27.511188 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 30 01:42:27.511188 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 30 01:42:27.511188 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Aug 30 01:42:27.511188 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Aug 30 01:42:27.515565 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 79/158] 10.222.0.27 () {56 vars in 1114 bytes} [Tue Aug 30 01:42:27 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 244 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 01:42:27.517403 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d204d73d-ca2e-4182-9055-7faf85c4b9d3 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:27.517848 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d204d73d-ca2e-4182-9055-7faf85c4b9d3 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:27.518245 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d204d73d-ca2e-4182-9055-7faf85c4b9d3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:27.530786 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:42:27.564817 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-d204d73d-ca2e-4182-9055-7faf85c4b9d3 None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 01:42:27.566304 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-d204d73d-ca2e-4182-9055-7faf85c4b9d3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:42:27.584239 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:42:27.594312 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34a4ac99-4441-4691-9ee7-606f8241d198 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:42:27.596893 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34a4ac99-4441-4691-9ee7-606f8241d198 None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:27.597375 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34a4ac99-4441-4691-9ee7-606f8241d198 None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:27.598891 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34a4ac99-4441-4691-9ee7-606f8241d198 None admin] PATH_INFO: `/v3/roles` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:27.599766 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34a4ac99-4441-4691-9ee7-606f8241d198 None admin] RBAC: Adding query filter params (name=reader) {{(pid=81578) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:42:27.603495 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34a4ac99-4441-4691-9ee7-606f8241d198 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:42:27.605223 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34a4ac99-4441-4691-9ee7-606f8241d198 None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:42:27.630543 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 80/159] 10.222.0.27 () {56 vars in 1101 bytes} [Tue Aug 30 01:42:27 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 348 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:42:27.637997 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34a4ac99-4441-4691-9ee7-606f8241d198 None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:42:27.693217 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 80/160] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Aug 30 01:42:27 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 177 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 30 01:42:27.694513 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d204d73d-ca2e-4182-9055-7faf85c4b9d3 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:42:27.848679 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34a4ac99-4441-4691-9ee7-606f8241d198 None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:42:27.902138 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d204d73d-ca2e-4182-9055-7faf85c4b9d3 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:42:27.923650 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f120be42-fce7-4b0b-a468-97b8745bc022 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:42:27.924367 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f120be42-fce7-4b0b-a468-97b8745bc022 None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:27.924719 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f120be42-fce7-4b0b-a468-97b8745bc022 None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:27.925067 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f120be42-fce7-4b0b-a468-97b8745bc022 None admin] PATH_INFO: `/v3/roles` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:27.925841 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f120be42-fce7-4b0b-a468-97b8745bc022 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81578) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:42:27.926219 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f120be42-fce7-4b0b-a468-97b8745bc022 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:42:27.927791 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f120be42-fce7-4b0b-a468-97b8745bc022 None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:42:27.942939 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aff692db-b018-4615-baf9-0d33a42f0f52 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:42:27.944325 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aff692db-b018-4615-baf9-0d33a42f0f52 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:27.944787 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aff692db-b018-4615-baf9-0d33a42f0f52 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:27.945389 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aff692db-b018-4615-baf9-0d33a42f0f52 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:27.951054 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 81/161] 10.222.0.27 () {56 vars in 1111 bytes} [Tue Aug 30 01:42:27 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 308 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:42:27.951054 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f120be42-fce7-4b0b-a468-97b8745bc022 None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:42:27.965415 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aff692db-b018-4615-baf9-0d33a42f0f52 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 30 01:42:27.979748 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application [None req-aff692db-b018-4615-baf9-0d33a42f0f52 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Aug 30 01:42:27.979748 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 01:42:27.979748 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 01:42:27.979748 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 01:42:27.979748 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 01:42:27.979748 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 01:42:27.979748 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 01:42:27.979748 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 01:42:27.979748 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 01:42:27.979748 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 01:42:27.979748 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 01:42:27.979748 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 01:42:27.979748 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 30 01:42:27.979748 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 30 01:42:27.979748 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 30 01:42:27.979748 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 30 01:42:27.981392 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 30 01:42:27.981392 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 30 01:42:27.981392 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 30 01:42:27.981392 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 30 01:42:27.981392 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 01:42:27.981392 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 01:42:27.981392 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 01:42:27.981392 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 01:42:27.981392 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 01:42:27.981392 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 01:42:27.981392 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 01:42:27.981392 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application with Lock( Aug 30 01:42:27.981392 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 01:42:27.981392 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._enter() Aug 30 01:42:27.981392 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 01:42:27.981392 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 01:42:27.982947 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 01:42:27.982947 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.creator() Aug 30 01:42:27.982947 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 01:42:27.982947 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application created_value = creator( Aug 30 01:42:27.982947 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 30 01:42:27.982947 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 30 01:42:27.982947 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 30 01:42:27.982947 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 30 01:42:27.982947 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 30 01:42:27.982947 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 30 01:42:27.982947 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Aug 30 01:42:27.982947 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Aug 30 01:42:27.994045 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 81/162] 10.222.0.27 () {56 vars in 1099 bytes} [Tue Aug 30 01:42:27 2022] GET /identity/v3/roles/member => generated 84 bytes in 295 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 01:42:27.996055 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aff692db-b018-4615-baf9-0d33a42f0f52 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:42:28.014138 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:42:28.082528 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:42:28.136196 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aff692db-b018-4615-baf9-0d33a42f0f52 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:42:28.187068 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7393602e-b883-42f2-b2bc-88f6956bf455 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:42:28.187716 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7393602e-b883-42f2-b2bc-88f6956bf455 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:28.187998 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7393602e-b883-42f2-b2bc-88f6956bf455 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:28.188271 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7393602e-b883-42f2-b2bc-88f6956bf455 None admin] PATH_INFO: `/v3/users/fac663577ae547588ada9d16b2fd6f5f` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:28.192132 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7393602e-b883-42f2-b2bc-88f6956bf455 None admin] RBAC: Authorizing `identity:get_user(user_id=fac663577ae547588ada9d16b2fd6f5f)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:42:28.193640 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7393602e-b883-42f2-b2bc-88f6956bf455 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:42:28.195250 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 82/163] 10.222.0.27 () {56 vars in 1177 bytes} [Tue Aug 30 01:42:27 2022] GET /identity/v3/users/fac663577ae547588ada9d16b2fd6f5f => generated 275 bytes in 200 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:42:28.196014 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7393602e-b883-42f2-b2bc-88f6956bf455 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:42:28.232189 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f120be42-fce7-4b0b-a468-97b8745bc022 None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:42:28.304352 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16d75efe-016f-4553-84fc-9ea204d1e2b6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:42:28.305483 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16d75efe-016f-4553-84fc-9ea204d1e2b6 None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:28.306039 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16d75efe-016f-4553-84fc-9ea204d1e2b6 None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:28.306845 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16d75efe-016f-4553-84fc-9ea204d1e2b6 None admin] PATH_INFO: `/v3/users/c9207557429f4320ba488c1bffb49e5b` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:28.311307 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16d75efe-016f-4553-84fc-9ea204d1e2b6 None admin] RBAC: Authorizing `identity:get_user(user_id=c9207557429f4320ba488c1bffb49e5b)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:42:28.314378 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16d75efe-016f-4553-84fc-9ea204d1e2b6 None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:42:28.316691 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 82/164] 10.222.0.27 () {56 vars in 1177 bytes} [Tue Aug 30 01:42:27 2022] GET /identity/v3/users/c9207557429f4320ba488c1bffb49e5b => generated 289 bytes in 370 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:42:28.325813 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16d75efe-016f-4553-84fc-9ea204d1e2b6 None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:42:28.339779 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:42:28.369405 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7393602e-b883-42f2-b2bc-88f6956bf455 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:42:28.424666 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba53e9d7-2831-48a6-902e-7c7e8c34d9a7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:42:28.425237 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba53e9d7-2831-48a6-902e-7c7e8c34d9a7 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:28.425545 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba53e9d7-2831-48a6-902e-7c7e8c34d9a7 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:28.425819 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba53e9d7-2831-48a6-902e-7c7e8c34d9a7 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:28.447196 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba53e9d7-2831-48a6-902e-7c7e8c34d9a7 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 30 01:42:28.452299 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application [None req-ba53e9d7-2831-48a6-902e-7c7e8c34d9a7 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Aug 30 01:42:28.452299 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 01:42:28.452299 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 01:42:28.452299 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 01:42:28.452299 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 01:42:28.452299 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 01:42:28.452299 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 01:42:28.452299 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 01:42:28.452299 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 01:42:28.452299 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 01:42:28.452299 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 01:42:28.452299 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 01:42:28.452299 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 30 01:42:28.452299 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 30 01:42:28.452299 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 30 01:42:28.452299 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 30 01:42:28.453202 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 30 01:42:28.453202 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 30 01:42:28.453202 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 30 01:42:28.453202 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 30 01:42:28.453202 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 01:42:28.453202 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 01:42:28.453202 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 01:42:28.453202 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 01:42:28.453202 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 01:42:28.453202 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 01:42:28.453202 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 01:42:28.453202 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application with Lock( Aug 30 01:42:28.453202 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 01:42:28.453202 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._enter() Aug 30 01:42:28.453202 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 01:42:28.453202 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 01:42:28.454084 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 01:42:28.454084 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.creator() Aug 30 01:42:28.454084 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 01:42:28.454084 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application created_value = creator( Aug 30 01:42:28.454084 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 30 01:42:28.454084 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 30 01:42:28.454084 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 30 01:42:28.454084 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 30 01:42:28.454084 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 30 01:42:28.454084 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 30 01:42:28.454084 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Aug 30 01:42:28.454084 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Aug 30 01:42:28.454084 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 83/165] 10.222.0.27 () {56 vars in 1099 bytes} [Tue Aug 30 01:42:28 2022] GET /identity/v3/roles/member => generated 84 bytes in 258 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 01:42:28.454756 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba53e9d7-2831-48a6-902e-7c7e8c34d9a7 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:42:28.534805 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16d75efe-016f-4553-84fc-9ea204d1e2b6 None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:42:28.548731 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:42:28.614204 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91c29425-cd34-47e5-b264-b1483c1ef031 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:42:28.614863 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91c29425-cd34-47e5-b264-b1483c1ef031 None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:28.615143 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91c29425-cd34-47e5-b264-b1483c1ef031 None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:28.615411 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91c29425-cd34-47e5-b264-b1483c1ef031 None admin] PATH_INFO: `/v3/roles` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:28.616084 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91c29425-cd34-47e5-b264-b1483c1ef031 None admin] RBAC: Adding query filter params (name=member) {{(pid=81578) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:42:28.616387 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91c29425-cd34-47e5-b264-b1483c1ef031 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:42:28.617999 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91c29425-cd34-47e5-b264-b1483c1ef031 None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:42:28.636086 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 83/166] 10.222.0.27 () {56 vars in 1101 bytes} [Tue Aug 30 01:42:28 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 319 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:42:28.636945 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91c29425-cd34-47e5-b264-b1483c1ef031 None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:42:28.656784 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba53e9d7-2831-48a6-902e-7c7e8c34d9a7 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:42:28.721857 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74d4b08d-486f-4faf-9428-e58a3bf3d1e9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:42:28.722556 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74d4b08d-486f-4faf-9428-e58a3bf3d1e9 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:28.722741 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74d4b08d-486f-4faf-9428-e58a3bf3d1e9 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:28.723005 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74d4b08d-486f-4faf-9428-e58a3bf3d1e9 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:28.732302 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:42:28.834237 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91c29425-cd34-47e5-b264-b1483c1ef031 None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:42:28.879727 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74d4b08d-486f-4faf-9428-e58a3bf3d1e9 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:42:28.881229 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74d4b08d-486f-4faf-9428-e58a3bf3d1e9 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:42:28.888223 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c41e3ef2-722d-4fe9-bb5d-6f20747127ef None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:42:28.888869 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c41e3ef2-722d-4fe9-bb5d-6f20747127ef None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:28.889145 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c41e3ef2-722d-4fe9-bb5d-6f20747127ef None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:28.889409 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c41e3ef2-722d-4fe9-bb5d-6f20747127ef None admin] PATH_INFO: `/v3/projects/90818f998d5f4520971eacf6ccc830ce` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:28.895265 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c41e3ef2-722d-4fe9-bb5d-6f20747127ef None admin] RBAC: Authorizing `identity:get_project(project_id=90818f998d5f4520971eacf6ccc830ce)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:42:28.896740 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c41e3ef2-722d-4fe9-bb5d-6f20747127ef None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:42:28.898517 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 84/167] 10.222.0.27 () {56 vars in 1186 bytes} [Tue Aug 30 01:42:28 2022] GET /identity/v3/projects/90818f998d5f4520971eacf6ccc830ce => generated 298 bytes in 262 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:42:28.899295 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c41e3ef2-722d-4fe9-bb5d-6f20747127ef None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:42:28.998159 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:42:29.012389 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 84/168] 10.222.0.27 () {58 vars in 1282 bytes} [Tue Aug 30 01:42:28 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 558 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 30 01:42:29.013258 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74d4b08d-486f-4faf-9428-e58a3bf3d1e9 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:42:29.052197 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:42:29.129696 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74d4b08d-486f-4faf-9428-e58a3bf3d1e9 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:42:29.170741 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4b6873f-4071-424a-9496-d38a65ba7bcf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:42:29.171749 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4b6873f-4071-424a-9496-d38a65ba7bcf None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:29.172250 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4b6873f-4071-424a-9496-d38a65ba7bcf None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:29.172877 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4b6873f-4071-424a-9496-d38a65ba7bcf None admin] PATH_INFO: `/v3/users/fac663577ae547588ada9d16b2fd6f5f` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:29.174849 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4b6873f-4071-424a-9496-d38a65ba7bcf None admin] RBAC: Authorizing `identity:get_user(user_id=fac663577ae547588ada9d16b2fd6f5f)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:42:29.176547 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4b6873f-4071-424a-9496-d38a65ba7bcf None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:42:29.178876 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 85/169] 10.222.0.27 () {56 vars in 1177 bytes} [Tue Aug 30 01:42:29 2022] GET /identity/v3/users/fac663577ae547588ada9d16b2fd6f5f => generated 275 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:42:29.180024 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4b6873f-4071-424a-9496-d38a65ba7bcf None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:42:29.187539 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c41e3ef2-722d-4fe9-bb5d-6f20747127ef None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:42:29.299156 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4b6873f-4071-424a-9496-d38a65ba7bcf None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:42:29.303460 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:42:29.314232 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e2ed4a8-68fa-4e22-b113-264e1e2b05e6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:42:29.320964 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e2ed4a8-68fa-4e22-b113-264e1e2b05e6 None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:29.320964 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e2ed4a8-68fa-4e22-b113-264e1e2b05e6 None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:29.320964 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e2ed4a8-68fa-4e22-b113-264e1e2b05e6 None admin] PATH_INFO: `/v3/roles` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:29.320964 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e2ed4a8-68fa-4e22-b113-264e1e2b05e6 None admin] RBAC: Adding query filter params (name=member) {{(pid=81578) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:42:29.320964 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e2ed4a8-68fa-4e22-b113-264e1e2b05e6 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:42:29.320964 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e2ed4a8-68fa-4e22-b113-264e1e2b05e6 None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:42:29.340136 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 85/170] 10.222.0.27 () {56 vars in 1101 bytes} [Tue Aug 30 01:42:28 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 441 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:42:29.341341 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e2ed4a8-68fa-4e22-b113-264e1e2b05e6 None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:42:29.346653 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6c1863f-a5d8-4360-931d-1aa6eed3643c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:42:29.347333 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6c1863f-a5d8-4360-931d-1aa6eed3643c None admin] REQUEST_METHOD: `PUT` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:29.347617 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6c1863f-a5d8-4360-931d-1aa6eed3643c None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:29.347893 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6c1863f-a5d8-4360-931d-1aa6eed3643c None admin] PATH_INFO: `/v3/projects/90818f998d5f4520971eacf6ccc830ce/users/c9207557429f4320ba488c1bffb49e5b/roles/ffde3a55b5dd43149c18c72b7f2fe6ae` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:29.353502 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6c1863f-a5d8-4360-931d-1aa6eed3643c None admin] RBAC: Authorizing `identity:create_grant(project_id=90818f998d5f4520971eacf6ccc830ce, user_id=c9207557429f4320ba488c1bffb49e5b, role_id=ffde3a55b5dd43149c18c72b7f2fe6ae)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:42:29.353502 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6c1863f-a5d8-4360-931d-1aa6eed3643c None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:42:29.433933 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 86/171] 10.222.0.27 () {58 vars in 1439 bytes} [Tue Aug 30 01:42:29 2022] PUT /identity/v3/projects/90818f998d5f4520971eacf6ccc830ce/users/c9207557429f4320ba488c1bffb49e5b/roles/ffde3a55b5dd43149c18c72b7f2fe6ae => generated 0 bytes in 254 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 01:42:29.434794 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6c1863f-a5d8-4360-931d-1aa6eed3643c None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:42:29.461208 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:42:29.466997 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e2ed4a8-68fa-4e22-b113-264e1e2b05e6 None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:42:29.510406 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ade7cde4-af3c-4b86-a772-38ede1e1e3d5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:42:29.511723 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ade7cde4-af3c-4b86-a772-38ede1e1e3d5 None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:29.512329 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ade7cde4-af3c-4b86-a772-38ede1e1e3d5 None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:29.512849 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ade7cde4-af3c-4b86-a772-38ede1e1e3d5 None admin] PATH_INFO: `/v3/projects/90818f998d5f4520971eacf6ccc830ce` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:29.514506 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ade7cde4-af3c-4b86-a772-38ede1e1e3d5 None admin] RBAC: Authorizing `identity:get_project(project_id=90818f998d5f4520971eacf6ccc830ce)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:42:29.516311 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ade7cde4-af3c-4b86-a772-38ede1e1e3d5 None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:42:29.518381 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 86/172] 10.222.0.27 () {56 vars in 1186 bytes} [Tue Aug 30 01:42:29 2022] GET /identity/v3/projects/90818f998d5f4520971eacf6ccc830ce => generated 298 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:42:29.519583 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ade7cde4-af3c-4b86-a772-38ede1e1e3d5 None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:42:29.599371 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6c1863f-a5d8-4360-931d-1aa6eed3643c None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:42:29.643086 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81c72390-dfc0-43ba-93d2-57740d71886d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:42:29.643616 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81c72390-dfc0-43ba-93d2-57740d71886d None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:29.643900 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81c72390-dfc0-43ba-93d2-57740d71886d None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:29.644172 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81c72390-dfc0-43ba-93d2-57740d71886d None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:29.653599 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ade7cde4-af3c-4b86-a772-38ede1e1e3d5 None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:42:29.677760 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:42:29.695177 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-247f2608-ee15-4ba5-90e6-93cc7801c640 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:42:29.695823 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-247f2608-ee15-4ba5-90e6-93cc7801c640 None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:29.696098 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-247f2608-ee15-4ba5-90e6-93cc7801c640 None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:29.696937 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-247f2608-ee15-4ba5-90e6-93cc7801c640 None admin] PATH_INFO: `/v3/users/fac663577ae547588ada9d16b2fd6f5f` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:29.698165 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-247f2608-ee15-4ba5-90e6-93cc7801c640 None admin] RBAC: Authorizing `identity:get_user(user_id=fac663577ae547588ada9d16b2fd6f5f)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:42:29.699624 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-247f2608-ee15-4ba5-90e6-93cc7801c640 None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:42:29.701487 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 87/173] 10.222.0.27 () {56 vars in 1177 bytes} [Tue Aug 30 01:42:29 2022] GET /identity/v3/users/fac663577ae547588ada9d16b2fd6f5f => generated 275 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:42:29.707562 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca5752b2-7674-4f4b-8789-3ca503238acd None None] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:29.708360 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca5752b2-7674-4f4b-8789-3ca503238acd None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:29.708836 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca5752b2-7674-4f4b-8789-3ca503238acd None None] PATH_INFO: `/` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:29.711289 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 88/174] 10.222.0.27 () {54 vars in 927 bytes} [Tue Aug 30 01:42:29 2022] GET /identity => generated 270 bytes in 9 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 01:42:29.712763 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28ff80e6-464c-49f9-8d40-23ada81db5ed None None] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:29.713523 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28ff80e6-464c-49f9-8d40-23ada81db5ed None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:29.713818 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28ff80e6-464c-49f9-8d40-23ada81db5ed None None] PATH_INFO: `/` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:29.715213 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 89/175] 10.222.0.27 () {54 vars in 927 bytes} [Tue Aug 30 01:42:29 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 01:42:29.716179 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28ff80e6-464c-49f9-8d40-23ada81db5ed None None] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:42:29.725402 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81c72390-dfc0-43ba-93d2-57740d71886d None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:42:29.727690 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81c72390-dfc0-43ba-93d2-57740d71886d None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:42:29.754809 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:42:29.831392 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28ff80e6-464c-49f9-8d40-23ada81db5ed None None] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:42:29.842771 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 87/176] 10.222.0.27 () {58 vars in 1282 bytes} [Tue Aug 30 01:42:29 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 403 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 30 01:42:29.842771 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4932afe6-175a-4eb9-9569-36dc5bae3ae4 None None] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:29.842771 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4932afe6-175a-4eb9-9569-36dc5bae3ae4 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:29.842771 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4932afe6-175a-4eb9-9569-36dc5bae3ae4 None None] PATH_INFO: `/` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:29.842771 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 88/177] 10.222.0.27 () {54 vars in 927 bytes} [Tue Aug 30 01:42:29 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 01:42:29.842771 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4932afe6-175a-4eb9-9569-36dc5bae3ae4 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:42:29.874171 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bc3e911-8c2f-4fa7-baf9-0ae09e734eed None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:42:29.874696 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bc3e911-8c2f-4fa7-baf9-0ae09e734eed None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:29.874975 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bc3e911-8c2f-4fa7-baf9-0ae09e734eed None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:29.875244 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bc3e911-8c2f-4fa7-baf9-0ae09e734eed None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:29.875975 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3bc3e911-8c2f-4fa7-baf9-0ae09e734eed None admin] RBAC: Adding query filter params (role.id=4621068972ba4f79bad104f956db04aa, scope.project.id=90818f998d5f4520971eacf6ccc830ce, user.id=fac663577ae547588ada9d16b2fd6f5f) {{(pid=81578) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:42:29.876284 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3bc3e911-8c2f-4fa7-baf9-0ae09e734eed None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:42:29.877744 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3bc3e911-8c2f-4fa7-baf9-0ae09e734eed None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:42:29.884646 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 90/178] 10.222.0.27 () {56 vars in 1374 bytes} [Tue Aug 30 01:42:29 2022] GET /identity/v3/role_assignments?user.id=fac663577ae547588ada9d16b2fd6f5f&scope.project.id=90818f998d5f4520971eacf6ccc830ce&role.id=4621068972ba4f79bad104f956db04aa => generated 608 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:42:29.885939 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91ebdf20-b4e0-4ee5-bf8f-0f5f8658910c None None] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:29.886143 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91ebdf20-b4e0-4ee5-bf8f-0f5f8658910c None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:29.886412 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91ebdf20-b4e0-4ee5-bf8f-0f5f8658910c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:29.908385 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.auth.core [None req-91ebdf20-b4e0-4ee5-bf8f-0f5f8658910c None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81578) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 01:42:29.909743 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-91ebdf20-b4e0-4ee5-bf8f-0f5f8658910c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:42:29.931513 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:42:29.989590 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4932afe6-175a-4eb9-9569-36dc5bae3ae4 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:42:30.032228 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 91/179] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Aug 30 01:42:29 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 147 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 30 01:42:30.034397 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70752cbe-d731-441f-9e44-3a97662153f5 None None] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:30.034902 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70752cbe-d731-441f-9e44-3a97662153f5 None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:30.035378 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70752cbe-d731-441f-9e44-3a97662153f5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:30.037160 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7766357c-bf95-4eff-8d68-668f7520a1c0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:42:30.037854 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7766357c-bf95-4eff-8d68-668f7520a1c0 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:30.038131 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7766357c-bf95-4eff-8d68-668f7520a1c0 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:30.038406 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7766357c-bf95-4eff-8d68-668f7520a1c0 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:30.058581 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.auth.core [None req-70752cbe-d731-441f-9e44-3a97662153f5 None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81578) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 01:42:30.060245 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-70752cbe-d731-441f-9e44-3a97662153f5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:42:30.090348 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:42:30.132103 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7766357c-bf95-4eff-8d68-668f7520a1c0 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:42:30.133573 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7766357c-bf95-4eff-8d68-668f7520a1c0 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:42:30.155393 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 92/180] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Aug 30 01:42:30 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 122 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 30 01:42:30.156591 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70752cbe-d731-441f-9e44-3a97662153f5 None None] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:42:30.209121 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:42:30.256408 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 89/181] 10.222.0.27 () {58 vars in 1282 bytes} [Tue Aug 30 01:42:29 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 415 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 30 01:42:30.261608 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec7674b2-cd09-410c-8231-b105c5637617 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:30.261983 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec7674b2-cd09-410c-8231-b105c5637617 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:30.262264 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec7674b2-cd09-410c-8231-b105c5637617 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:30.284253 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-ec7674b2-cd09-410c-8231-b105c5637617 None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 01:42:30.285586 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-ec7674b2-cd09-410c-8231-b105c5637617 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:42:30.298699 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70752cbe-d731-441f-9e44-3a97662153f5 None None] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:42:30.344711 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa94f389-5a4f-4705-9f48-089c3f6aa324 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:42:30.345406 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa94f389-5a4f-4705-9f48-089c3f6aa324 None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:30.345723 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa94f389-5a4f-4705-9f48-089c3f6aa324 None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:30.345994 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa94f389-5a4f-4705-9f48-089c3f6aa324 None admin] PATH_INFO: `/v3/projects/2b886b3234b7472ca4968ccca426a61d` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:30.349809 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa94f389-5a4f-4705-9f48-089c3f6aa324 None admin] RBAC: Authorizing `identity:get_project(project_id=2b886b3234b7472ca4968ccca426a61d)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:42:30.351519 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa94f389-5a4f-4705-9f48-089c3f6aa324 None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:42:30.354142 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 93/182] 10.222.0.27 () {56 vars in 1186 bytes} [Tue Aug 30 01:42:30 2022] GET /identity/v3/projects/2b886b3234b7472ca4968ccca426a61d => generated 312 bytes in 198 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:42:30.354773 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa94f389-5a4f-4705-9f48-089c3f6aa324 None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:42:30.368756 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 90/183] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Aug 30 01:42:30 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 111 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 30 01:42:30.370586 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec7674b2-cd09-410c-8231-b105c5637617 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:42:30.402757 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:42:30.481938 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:42:30.492945 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec7674b2-cd09-410c-8231-b105c5637617 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:42:30.542624 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3142a715-f3ed-4368-81f9-c37f186f1f6e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:42:30.543649 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3142a715-f3ed-4368-81f9-c37f186f1f6e None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:30.545227 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3142a715-f3ed-4368-81f9-c37f186f1f6e None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:30.546925 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3142a715-f3ed-4368-81f9-c37f186f1f6e None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:30.547868 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3142a715-f3ed-4368-81f9-c37f186f1f6e None admin] RBAC: Adding query filter params (role.id=1fb81afcb9f245f3b989add94598a063, scope.project.id=2b886b3234b7472ca4968ccca426a61d, user.id=fac663577ae547588ada9d16b2fd6f5f) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:42:30.549225 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3142a715-f3ed-4368-81f9-c37f186f1f6e None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:42:30.550908 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3142a715-f3ed-4368-81f9-c37f186f1f6e None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:42:30.559904 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa94f389-5a4f-4705-9f48-089c3f6aa324 None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:42:30.565201 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 91/184] 10.222.0.27 () {56 vars in 1374 bytes} [Tue Aug 30 01:42:30 2022] GET /identity/v3/role_assignments?user.id=fac663577ae547588ada9d16b2fd6f5f&scope.project.id=2b886b3234b7472ca4968ccca426a61d&role.id=1fb81afcb9f245f3b989add94598a063 => generated 608 bytes in 195 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:42:30.567228 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3142a715-f3ed-4368-81f9-c37f186f1f6e None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:42:30.624705 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfbb2f5c-9805-4e6b-b9e3-9244cc780d57 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:42:30.627145 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfbb2f5c-9805-4e6b-b9e3-9244cc780d57 None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:30.627651 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfbb2f5c-9805-4e6b-b9e3-9244cc780d57 None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:30.629153 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfbb2f5c-9805-4e6b-b9e3-9244cc780d57 None admin] PATH_INFO: `/v3/projects/90818f998d5f4520971eacf6ccc830ce` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:30.631456 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cfbb2f5c-9805-4e6b-b9e3-9244cc780d57 None admin] RBAC: Authorizing `identity:get_project(project_id=90818f998d5f4520971eacf6ccc830ce)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:42:30.635971 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cfbb2f5c-9805-4e6b-b9e3-9244cc780d57 None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:42:30.645167 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 94/185] 10.222.0.27 () {56 vars in 1186 bytes} [Tue Aug 30 01:42:30 2022] GET /identity/v3/projects/90818f998d5f4520971eacf6ccc830ce => generated 298 bytes in 291 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:42:30.646394 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfbb2f5c-9805-4e6b-b9e3-9244cc780d57 None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:42:30.762693 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3142a715-f3ed-4368-81f9-c37f186f1f6e None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:42:30.793803 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:42:30.801281 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfbb2f5c-9805-4e6b-b9e3-9244cc780d57 None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:42:30.810195 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b60bc45a-ed79-4b7c-a7b1-a91afea94152 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:42:30.810838 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b60bc45a-ed79-4b7c-a7b1-a91afea94152 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:30.811117 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b60bc45a-ed79-4b7c-a7b1-a91afea94152 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:30.811388 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b60bc45a-ed79-4b7c-a7b1-a91afea94152 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:30.826263 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b60bc45a-ed79-4b7c-a7b1-a91afea94152 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 30 01:42:30.832813 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application [None req-b60bc45a-ed79-4b7c-a7b1-a91afea94152 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 30 01:42:30.832813 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 01:42:30.832813 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 01:42:30.832813 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 01:42:30.832813 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 01:42:30.832813 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 01:42:30.832813 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 01:42:30.832813 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 01:42:30.832813 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 01:42:30.832813 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 01:42:30.832813 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 01:42:30.832813 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 01:42:30.832813 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 30 01:42:30.832813 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 30 01:42:30.832813 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 30 01:42:30.832813 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 30 01:42:30.834181 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 30 01:42:30.834181 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 30 01:42:30.834181 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 30 01:42:30.834181 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 30 01:42:30.834181 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 01:42:30.834181 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 01:42:30.834181 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 01:42:30.834181 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 01:42:30.834181 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 01:42:30.834181 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 01:42:30.834181 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 01:42:30.834181 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application with Lock( Aug 30 01:42:30.834181 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 01:42:30.834181 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._enter() Aug 30 01:42:30.834181 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 01:42:30.834181 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 01:42:30.834992 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 01:42:30.834992 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.creator() Aug 30 01:42:30.834992 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 01:42:30.834992 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application created_value = creator( Aug 30 01:42:30.834992 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 30 01:42:30.834992 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 30 01:42:30.834992 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 30 01:42:30.834992 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 30 01:42:30.834992 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 30 01:42:30.834992 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 30 01:42:30.834992 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 30 01:42:30.834992 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Aug 30 01:42:30.834992 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 92/186] 10.222.0.27 () {56 vars in 1096 bytes} [Tue Aug 30 01:42:30 2022] GET /identity/v3/roles/admin => generated 83 bytes in 267 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 01:42:30.834992 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b60bc45a-ed79-4b7c-a7b1-a91afea94152 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:42:30.846402 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fad8dd24-ae92-4d9e-897a-fa9a56d6d6e3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:42:30.847813 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fad8dd24-ae92-4d9e-897a-fa9a56d6d6e3 None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:30.848313 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fad8dd24-ae92-4d9e-897a-fa9a56d6d6e3 None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:30.848792 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fad8dd24-ae92-4d9e-897a-fa9a56d6d6e3 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:30.850402 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fad8dd24-ae92-4d9e-897a-fa9a56d6d6e3 None admin] RBAC: Adding query filter params (role.id=1fb81afcb9f245f3b989add94598a063, scope.project.id=90818f998d5f4520971eacf6ccc830ce, user.id=fac663577ae547588ada9d16b2fd6f5f) {{(pid=81578) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:42:30.850889 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fad8dd24-ae92-4d9e-897a-fa9a56d6d6e3 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:42:30.854126 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fad8dd24-ae92-4d9e-897a-fa9a56d6d6e3 None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:42:30.857625 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:42:30.862031 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 95/187] 10.222.0.27 () {56 vars in 1374 bytes} [Tue Aug 30 01:42:30 2022] GET /identity/v3/role_assignments?user.id=fac663577ae547588ada9d16b2fd6f5f&scope.project.id=90818f998d5f4520971eacf6ccc830ce&role.id=1fb81afcb9f245f3b989add94598a063 => generated 608 bytes in 216 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:42:30.863587 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fad8dd24-ae92-4d9e-897a-fa9a56d6d6e3 None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:42:30.983757 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b60bc45a-ed79-4b7c-a7b1-a91afea94152 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:42:31.008302 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fad8dd24-ae92-4d9e-897a-fa9a56d6d6e3 None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:42:31.021571 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c743384b-4314-4cdf-934a-e7f198a9b292 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:42:31.022329 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c743384b-4314-4cdf-934a-e7f198a9b292 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:31.022681 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c743384b-4314-4cdf-934a-e7f198a9b292 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:31.023032 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c743384b-4314-4cdf-934a-e7f198a9b292 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:31.037653 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c743384b-4314-4cdf-934a-e7f198a9b292 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 30 01:42:31.043883 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application [None req-c743384b-4314-4cdf-934a-e7f198a9b292 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Aug 30 01:42:31.043883 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 01:42:31.043883 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 01:42:31.043883 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 01:42:31.043883 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 01:42:31.043883 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 01:42:31.043883 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 01:42:31.043883 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 01:42:31.043883 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 01:42:31.043883 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 01:42:31.043883 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 01:42:31.043883 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 01:42:31.043883 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 30 01:42:31.043883 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 30 01:42:31.043883 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 30 01:42:31.043883 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 30 01:42:31.044848 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 30 01:42:31.044848 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 30 01:42:31.044848 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 30 01:42:31.044848 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 30 01:42:31.044848 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 01:42:31.044848 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 01:42:31.044848 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 01:42:31.044848 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 01:42:31.044848 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 01:42:31.044848 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 01:42:31.044848 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 01:42:31.044848 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application with Lock( Aug 30 01:42:31.044848 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 01:42:31.044848 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._enter() Aug 30 01:42:31.044848 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 01:42:31.044848 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 01:42:31.045973 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 01:42:31.045973 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.creator() Aug 30 01:42:31.045973 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 01:42:31.045973 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application created_value = creator( Aug 30 01:42:31.045973 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 30 01:42:31.045973 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 30 01:42:31.045973 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 30 01:42:31.045973 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 30 01:42:31.045973 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 30 01:42:31.045973 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 30 01:42:31.045973 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Aug 30 01:42:31.045973 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Aug 30 01:42:31.045973 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 93/188] 10.222.0.27 () {56 vars in 1114 bytes} [Tue Aug 30 01:42:30 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 211 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 01:42:31.045973 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c743384b-4314-4cdf-934a-e7f198a9b292 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:42:31.062298 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2a1ca06-397a-4532-aa8d-3da2563cc35e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:42:31.063030 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2a1ca06-397a-4532-aa8d-3da2563cc35e None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:31.063379 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2a1ca06-397a-4532-aa8d-3da2563cc35e None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:31.063735 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2a1ca06-397a-4532-aa8d-3da2563cc35e None admin] PATH_INFO: `/v3/domains/default` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:31.065049 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2a1ca06-397a-4532-aa8d-3da2563cc35e None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:42:31.066800 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2a1ca06-397a-4532-aa8d-3da2563cc35e None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:42:31.068311 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 96/189] 10.222.0.27 () {56 vars in 1108 bytes} [Tue Aug 30 01:42:30 2022] GET /identity/v3/domains/default => generated 199 bytes in 205 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:42:31.069069 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2a1ca06-397a-4532-aa8d-3da2563cc35e None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:42:31.104119 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:42:31.157847 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:42:31.191576 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c743384b-4314-4cdf-934a-e7f198a9b292 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:42:31.199373 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2a1ca06-397a-4532-aa8d-3da2563cc35e None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:42:31.228826 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abdf9125-7467-4356-a44d-6d8912543c85 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:42:31.229832 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abdf9125-7467-4356-a44d-6d8912543c85 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:31.230112 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abdf9125-7467-4356-a44d-6d8912543c85 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:31.230379 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abdf9125-7467-4356-a44d-6d8912543c85 None admin] PATH_INFO: `/v3/roles` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:31.231079 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abdf9125-7467-4356-a44d-6d8912543c85 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:42:31.231383 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abdf9125-7467-4356-a44d-6d8912543c85 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:42:31.232783 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abdf9125-7467-4356-a44d-6d8912543c85 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:42:31.241777 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-629ef16d-99b2-4205-a538-8e2367eb239b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:42:31.242006 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 94/190] 10.222.0.27 () {56 vars in 1099 bytes} [Tue Aug 30 01:42:31 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 196 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:42:31.242765 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abdf9125-7467-4356-a44d-6d8912543c85 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:42:31.247021 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-629ef16d-99b2-4205-a538-8e2367eb239b None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:31.247021 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-629ef16d-99b2-4205-a538-8e2367eb239b None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:31.247266 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-629ef16d-99b2-4205-a538-8e2367eb239b None admin] PATH_INFO: `/v3/roles` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:31.247925 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-629ef16d-99b2-4205-a538-8e2367eb239b None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81578) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:42:31.248231 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-629ef16d-99b2-4205-a538-8e2367eb239b None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:42:31.249917 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-629ef16d-99b2-4205-a538-8e2367eb239b None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:42:31.258263 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 97/191] 10.222.0.27 () {56 vars in 1111 bytes} [Tue Aug 30 01:42:31 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 189 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:42:31.259516 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-629ef16d-99b2-4205-a538-8e2367eb239b None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:42:31.354358 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:42:31.365714 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abdf9125-7467-4356-a44d-6d8912543c85 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:42:31.367836 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-629ef16d-99b2-4205-a538-8e2367eb239b None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:42:31.404207 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12af69b6-d74c-4f66-9a19-fb25f70233b6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:42:31.404918 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12af69b6-d74c-4f66-9a19-fb25f70233b6 None admin] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:31.405204 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12af69b6-d74c-4f66-9a19-fb25f70233b6 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:31.405482 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ae2eca0-04f2-4b15-967c-8a9256e73c7e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:42:31.405603 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12af69b6-d74c-4f66-9a19-fb25f70233b6 None admin] PATH_INFO: `/v3/users` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:31.406133 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ae2eca0-04f2-4b15-967c-8a9256e73c7e None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:31.406231 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12af69b6-d74c-4f66-9a19-fb25f70233b6 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:42:31.406415 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ae2eca0-04f2-4b15-967c-8a9256e73c7e None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:31.406725 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ae2eca0-04f2-4b15-967c-8a9256e73c7e None admin] PATH_INFO: `/v3/users/a4a8afdab07d4fd089132965087a3d56` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:31.407615 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12af69b6-d74c-4f66-9a19-fb25f70233b6 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:42:31.423589 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:42:31.424666 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ae2eca0-04f2-4b15-967c-8a9256e73c7e None admin] RBAC: Authorizing `identity:get_user(user_id=a4a8afdab07d4fd089132965087a3d56)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:42:31.426180 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ae2eca0-04f2-4b15-967c-8a9256e73c7e None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:42:31.427987 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 98/192] 10.222.0.27 () {56 vars in 1177 bytes} [Tue Aug 30 01:42:31 2022] GET /identity/v3/users/a4a8afdab07d4fd089132965087a3d56 => generated 283 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:42:31.433861 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ae2eca0-04f2-4b15-967c-8a9256e73c7e None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:42:31.530059 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ae2eca0-04f2-4b15-967c-8a9256e73c7e None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:42:31.564427 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c5545d8-665b-49bb-b82a-ae990db18eb2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:42:31.565050 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c5545d8-665b-49bb-b82a-ae990db18eb2 None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:31.565324 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c5545d8-665b-49bb-b82a-ae990db18eb2 None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:31.565618 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c5545d8-665b-49bb-b82a-ae990db18eb2 None admin] PATH_INFO: `/v3/users/a4a8afdab07d4fd089132965087a3d56` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:31.566719 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c5545d8-665b-49bb-b82a-ae990db18eb2 None admin] RBAC: Authorizing `identity:get_user(user_id=a4a8afdab07d4fd089132965087a3d56)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:42:31.568146 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c5545d8-665b-49bb-b82a-ae990db18eb2 None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:42:31.569745 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 99/193] 10.222.0.27 () {56 vars in 1177 bytes} [Tue Aug 30 01:42:31 2022] GET /identity/v3/users/a4a8afdab07d4fd089132965087a3d56 => generated 283 bytes in 141 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:42:31.570543 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c5545d8-665b-49bb-b82a-ae990db18eb2 None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:42:31.585589 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:42:31.667127 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c5545d8-665b-49bb-b82a-ae990db18eb2 None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:42:31.701167 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76281afa-4fcf-4583-b35d-59fc058eb92a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:42:31.701811 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76281afa-4fcf-4583-b35d-59fc058eb92a None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:31.702088 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76281afa-4fcf-4583-b35d-59fc058eb92a None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:31.702357 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76281afa-4fcf-4583-b35d-59fc058eb92a None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:31.768130 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76281afa-4fcf-4583-b35d-59fc058eb92a None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:42:31.769607 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76281afa-4fcf-4583-b35d-59fc058eb92a None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:42:31.862956 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 100/194] 10.222.0.27 () {58 vars in 1282 bytes} [Tue Aug 30 01:42:31 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 293 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 30 01:42:31.864042 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76281afa-4fcf-4583-b35d-59fc058eb92a None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:42:31.945596 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 95/195] 10.222.0.27 () {60 vars in 1132 bytes} [Tue Aug 30 01:42:31 2022] POST /identity/v3/users => generated 297 bytes in 703 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 01:42:31.947734 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-973bc135-1998-4b3f-bc8f-4714c0478062 None None] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:31.948017 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-973bc135-1998-4b3f-bc8f-4714c0478062 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:31.948299 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-973bc135-1998-4b3f-bc8f-4714c0478062 None None] PATH_INFO: `/` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:31.953904 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 96/196] 10.222.0.27 () {54 vars in 927 bytes} [Tue Aug 30 01:42:31 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 01:42:31.953904 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-973bc135-1998-4b3f-bc8f-4714c0478062 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:42:31.964314 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:42:31.967994 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76281afa-4fcf-4583-b35d-59fc058eb92a None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:42:32.013489 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9aa11a2-0954-47a9-bc7b-ea9e4e3a1c75 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:42:32.014096 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9aa11a2-0954-47a9-bc7b-ea9e4e3a1c75 None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:32.014372 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9aa11a2-0954-47a9-bc7b-ea9e4e3a1c75 None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:32.014637 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9aa11a2-0954-47a9-bc7b-ea9e4e3a1c75 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:32.040803 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:42:32.072209 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-973bc135-1998-4b3f-bc8f-4714c0478062 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:42:32.107738 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9aa11a2-0954-47a9-bc7b-ea9e4e3a1c75 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:42:32.109232 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9aa11a2-0954-47a9-bc7b-ea9e4e3a1c75 None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:42:32.115037 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea9331d9-d64d-4f59-a728-16fbf69c4cf5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:42:32.115608 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea9331d9-d64d-4f59-a728-16fbf69c4cf5 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:32.115890 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea9331d9-d64d-4f59-a728-16fbf69c4cf5 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:32.116181 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea9331d9-d64d-4f59-a728-16fbf69c4cf5 None admin] PATH_INFO: `/v3/projects/67913d5682a443069a5eed96f179751f` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:32.117302 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea9331d9-d64d-4f59-a728-16fbf69c4cf5 None admin] RBAC: Authorizing `identity:get_project(project_id=67913d5682a443069a5eed96f179751f)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:42:32.119360 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea9331d9-d64d-4f59-a728-16fbf69c4cf5 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:42:32.121178 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 97/197] 10.222.0.27 () {56 vars in 1186 bytes} [Tue Aug 30 01:42:31 2022] GET /identity/v3/projects/67913d5682a443069a5eed96f179751f => generated 302 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:42:32.122238 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c602f3a8-0e88-43d2-9ab0-ce937a17a86c None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:32.122552 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c602f3a8-0e88-43d2-9ab0-ce937a17a86c None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:32.122864 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c602f3a8-0e88-43d2-9ab0-ce937a17a86c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:32.141899 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-c602f3a8-0e88-43d2-9ab0-ce937a17a86c None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 01:42:32.143448 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-c602f3a8-0e88-43d2-9ab0-ce937a17a86c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:42:32.225189 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:INSERT=1,keystone:SELECT=99 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:42:32.234365 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 101/198] 10.222.0.27 () {58 vars in 1282 bytes} [Tue Aug 30 01:42:31 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 371 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 30 01:42:32.235300 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9aa11a2-0954-47a9-bc7b-ea9e4e3a1c75 None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:42:32.240953 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 98/199] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Aug 30 01:42:32 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 119 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 30 01:42:32.242562 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c602f3a8-0e88-43d2-9ab0-ce937a17a86c None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:42:32.253046 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:42:32.354720 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9aa11a2-0954-47a9-bc7b-ea9e4e3a1c75 None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:42:32.375281 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c602f3a8-0e88-43d2-9ab0-ce937a17a86c None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:42:32.396808 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-713b55c0-d29f-4daa-9863-3486f7c209ce None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:42:32.397534 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-713b55c0-d29f-4daa-9863-3486f7c209ce None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:32.397882 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-713b55c0-d29f-4daa-9863-3486f7c209ce None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:32.398243 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-713b55c0-d29f-4daa-9863-3486f7c209ce None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:32.399042 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-713b55c0-d29f-4daa-9863-3486f7c209ce None admin] RBAC: Adding query filter params (role.id=ead5fac86a0747ea8823a82efcea8ed7, scope.project.id=67913d5682a443069a5eed96f179751f, user.id=a4a8afdab07d4fd089132965087a3d56) {{(pid=81578) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:42:32.399446 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-713b55c0-d29f-4daa-9863-3486f7c209ce None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:42:32.400953 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-713b55c0-d29f-4daa-9863-3486f7c209ce None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:42:32.411812 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 102/200] 10.222.0.27 () {56 vars in 1374 bytes} [Tue Aug 30 01:42:32 2022] GET /identity/v3/role_assignments?user.id=a4a8afdab07d4fd089132965087a3d56&scope.project.id=67913d5682a443069a5eed96f179751f&role.id=ead5fac86a0747ea8823a82efcea8ed7 => generated 259 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:42:32.422491 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db7c6112-e55a-4ff4-b240-ec65ac893b4e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:42:32.423281 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db7c6112-e55a-4ff4-b240-ec65ac893b4e None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:32.423679 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db7c6112-e55a-4ff4-b240-ec65ac893b4e None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:32.424019 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db7c6112-e55a-4ff4-b240-ec65ac893b4e None admin] PATH_INFO: `/v3/projects/67913d5682a443069a5eed96f179751f` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:32.425515 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db7c6112-e55a-4ff4-b240-ec65ac893b4e None admin] RBAC: Authorizing `identity:get_project(project_id=67913d5682a443069a5eed96f179751f)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:42:32.427108 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db7c6112-e55a-4ff4-b240-ec65ac893b4e None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:42:32.429147 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 99/201] 10.222.0.27 () {56 vars in 1186 bytes} [Tue Aug 30 01:42:32 2022] GET /identity/v3/projects/67913d5682a443069a5eed96f179751f => generated 302 bytes in 188 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:42:32.434898 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-713b55c0-d29f-4daa-9863-3486f7c209ce None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:42:32.523311 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:42:32.546276 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db7c6112-e55a-4ff4-b240-ec65ac893b4e None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:42:32.556823 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-713b55c0-d29f-4daa-9863-3486f7c209ce None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:42:32.623587 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf698666-1f11-465f-b03b-816f5566d32a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:42:32.624846 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf698666-1f11-465f-b03b-816f5566d32a None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:32.625228 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf698666-1f11-465f-b03b-816f5566d32a None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:32.625707 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf698666-1f11-465f-b03b-816f5566d32a None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:32.626521 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf698666-1f11-465f-b03b-816f5566d32a None admin] RBAC: Adding query filter params (role.id=4621068972ba4f79bad104f956db04aa, scope.project.id=67913d5682a443069a5eed96f179751f, user.id=a4a8afdab07d4fd089132965087a3d56) {{(pid=81578) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:42:32.626940 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf698666-1f11-465f-b03b-816f5566d32a None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:42:32.628633 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf698666-1f11-465f-b03b-816f5566d32a None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:42:32.637258 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 103/202] 10.222.0.27 () {56 vars in 1374 bytes} [Tue Aug 30 01:42:32 2022] GET /identity/v3/role_assignments?user.id=a4a8afdab07d4fd089132965087a3d56&scope.project.id=67913d5682a443069a5eed96f179751f&role.id=4621068972ba4f79bad104f956db04aa => generated 259 bytes in 203 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 30 01:42:32.647452 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:42:32.697315 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db7c6112-e55a-4ff4-b240-ec65ac893b4e None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:42:32.775843 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4278a1c7-e16f-4762-b9f2-21d27da45819 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:42:32.776563 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4278a1c7-e16f-4762-b9f2-21d27da45819 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:32.776846 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4278a1c7-e16f-4762-b9f2-21d27da45819 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:32.777119 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4278a1c7-e16f-4762-b9f2-21d27da45819 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:32.794248 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4278a1c7-e16f-4762-b9f2-21d27da45819 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 30 01:42:32.799853 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application [None req-4278a1c7-e16f-4762-b9f2-21d27da45819 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Aug 30 01:42:32.799853 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 01:42:32.799853 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 01:42:32.799853 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 01:42:32.799853 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 01:42:32.799853 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 01:42:32.799853 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 01:42:32.799853 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 01:42:32.799853 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 01:42:32.799853 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 01:42:32.799853 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 01:42:32.799853 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 01:42:32.799853 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 30 01:42:32.799853 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 30 01:42:32.799853 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 30 01:42:32.799853 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 30 01:42:32.801072 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 30 01:42:32.801072 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 30 01:42:32.801072 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 30 01:42:32.801072 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 30 01:42:32.801072 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 01:42:32.801072 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 01:42:32.801072 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 01:42:32.801072 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 01:42:32.801072 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 01:42:32.801072 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 01:42:32.801072 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 01:42:32.801072 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application with Lock( Aug 30 01:42:32.801072 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 01:42:32.801072 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._enter() Aug 30 01:42:32.801072 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 01:42:32.801072 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 01:42:32.802105 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 01:42:32.802105 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.creator() Aug 30 01:42:32.802105 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 01:42:32.802105 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application created_value = creator( Aug 30 01:42:32.802105 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 30 01:42:32.802105 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 30 01:42:32.802105 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 30 01:42:32.802105 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 30 01:42:32.802105 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 30 01:42:32.802105 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 30 01:42:32.802105 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Aug 30 01:42:32.802105 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Aug 30 01:42:32.802105 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 100/203] 10.222.0.27 () {56 vars in 1099 bytes} [Tue Aug 30 01:42:32 2022] GET /identity/v3/roles/reader => generated 84 bytes in 255 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 01:42:32.809180 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf698666-1f11-465f-b03b-816f5566d32a None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:42:32.946977 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf698666-1f11-465f-b03b-816f5566d32a None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:42:32.983260 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4433bb6-4f88-4dc2-b64b-8b06df996d5b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:42:32.984128 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4433bb6-4f88-4dc2-b64b-8b06df996d5b None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:32.984553 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4433bb6-4f88-4dc2-b64b-8b06df996d5b None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:32.984977 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4433bb6-4f88-4dc2-b64b-8b06df996d5b None admin] PATH_INFO: `/v3/roles` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:32.985888 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4433bb6-4f88-4dc2-b64b-8b06df996d5b None admin] RBAC: Adding query filter params (name=reader) {{(pid=81578) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:42:32.986359 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4433bb6-4f88-4dc2-b64b-8b06df996d5b None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:42:32.987868 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4433bb6-4f88-4dc2-b64b-8b06df996d5b None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:42:33.000638 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 104/204] 10.222.0.27 () {56 vars in 1101 bytes} [Tue Aug 30 01:42:32 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 192 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:42:33.006217 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4278a1c7-e16f-4762-b9f2-21d27da45819 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:42:33.143146 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4278a1c7-e16f-4762-b9f2-21d27da45819 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:42:33.172557 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:42:33.190280 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22e8186a-8d5b-461e-8a74-13638872c5f3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:42:33.190922 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22e8186a-8d5b-461e-8a74-13638872c5f3 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:33.191205 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22e8186a-8d5b-461e-8a74-13638872c5f3 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:33.191483 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22e8186a-8d5b-461e-8a74-13638872c5f3 None admin] PATH_INFO: `/v3/users/c9207557429f4320ba488c1bffb49e5b` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:33.198214 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22e8186a-8d5b-461e-8a74-13638872c5f3 None admin] RBAC: Authorizing `identity:get_user(user_id=c9207557429f4320ba488c1bffb49e5b)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:42:33.199625 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22e8186a-8d5b-461e-8a74-13638872c5f3 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:42:33.201607 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 101/205] 10.222.0.27 () {56 vars in 1177 bytes} [Tue Aug 30 01:42:33 2022] GET /identity/v3/users/c9207557429f4320ba488c1bffb49e5b => generated 289 bytes in 196 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:42:33.206922 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4433bb6-4f88-4dc2-b64b-8b06df996d5b None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:42:33.210758 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:42:33.327599 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4433bb6-4f88-4dc2-b64b-8b06df996d5b None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:42:33.378388 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17628adb-b5f1-48b0-86d9-a4800592e313 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:42:33.379117 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17628adb-b5f1-48b0-86d9-a4800592e313 None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:33.379406 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17628adb-b5f1-48b0-86d9-a4800592e313 None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:33.379689 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17628adb-b5f1-48b0-86d9-a4800592e313 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:33.470661 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17628adb-b5f1-48b0-86d9-a4800592e313 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:42:33.472062 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17628adb-b5f1-48b0-86d9-a4800592e313 None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:42:33.475079 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:42:33.612233 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 105/206] 10.222.0.27 () {58 vars in 1282 bytes} [Tue Aug 30 01:42:33 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 406 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 30 01:42:33.617441 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22e8186a-8d5b-461e-8a74-13638872c5f3 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:42:33.739534 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22e8186a-8d5b-461e-8a74-13638872c5f3 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:42:33.774663 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df72eba7-6e66-4cda-af0b-60af20078b29 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:42:33.775562 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df72eba7-6e66-4cda-af0b-60af20078b29 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:33.775844 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df72eba7-6e66-4cda-af0b-60af20078b29 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:33.776119 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df72eba7-6e66-4cda-af0b-60af20078b29 None admin] PATH_INFO: `/v3/projects/90818f998d5f4520971eacf6ccc830ce` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:33.777270 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df72eba7-6e66-4cda-af0b-60af20078b29 None admin] RBAC: Authorizing `identity:get_project(project_id=90818f998d5f4520971eacf6ccc830ce)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:42:33.778766 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df72eba7-6e66-4cda-af0b-60af20078b29 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:42:33.780406 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 102/207] 10.222.0.27 () {56 vars in 1186 bytes} [Tue Aug 30 01:42:33 2022] GET /identity/v3/projects/90818f998d5f4520971eacf6ccc830ce => generated 298 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:42:33.786330 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17628adb-b5f1-48b0-86d9-a4800592e313 None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:42:33.919922 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17628adb-b5f1-48b0-86d9-a4800592e313 None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:42:33.922711 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:42:33.956271 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79d05b33-2bca-48e2-944a-96e0c08637f6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:42:33.956916 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79d05b33-2bca-48e2-944a-96e0c08637f6 None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:33.957128 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79d05b33-2bca-48e2-944a-96e0c08637f6 None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:33.957392 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79d05b33-2bca-48e2-944a-96e0c08637f6 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:33.958142 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79d05b33-2bca-48e2-944a-96e0c08637f6 None admin] RBAC: Adding query filter params (role.id=ffde3a55b5dd43149c18c72b7f2fe6ae, scope.project.id=90818f998d5f4520971eacf6ccc830ce, user.id=c9207557429f4320ba488c1bffb49e5b) {{(pid=81578) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:42:33.958445 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79d05b33-2bca-48e2-944a-96e0c08637f6 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:42:33.959866 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79d05b33-2bca-48e2-944a-96e0c08637f6 None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:42:33.973829 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 106/208] 10.222.0.27 () {56 vars in 1374 bytes} [Tue Aug 30 01:42:33 2022] GET /identity/v3/role_assignments?user.id=c9207557429f4320ba488c1bffb49e5b&scope.project.id=90818f998d5f4520971eacf6ccc830ce&role.id=ffde3a55b5dd43149c18c72b7f2fe6ae => generated 608 bytes in 188 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:42:34.533495 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81da0604-c34a-461f-b997-be5bfec19ce2 None None] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:34.533949 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81da0604-c34a-461f-b997-be5bfec19ce2 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:34.534058 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81da0604-c34a-461f-b997-be5bfec19ce2 None None] PATH_INFO: `/` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:34.535426 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 103/209] 10.222.0.27 () {54 vars in 927 bytes} [Tue Aug 30 01:42:34 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 01:42:34.541441 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e53d91c2-373c-4102-9e49-7316ca45a7a3 None None] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:34.541758 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e53d91c2-373c-4102-9e49-7316ca45a7a3 None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:34.542195 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e53d91c2-373c-4102-9e49-7316ca45a7a3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:34.566037 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.auth.core [None req-e53d91c2-373c-4102-9e49-7316ca45a7a3 None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81578) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 01:42:34.567454 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-e53d91c2-373c-4102-9e49-7316ca45a7a3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:42:34.652391 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 107/210] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Aug 30 01:42:34 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 112 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 30 01:42:34.656053 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8eca2b1b-8e53-46d6-8476-fa00b8cfaabe None None] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:34.656053 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8eca2b1b-8e53-46d6-8476-fa00b8cfaabe None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:34.656053 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8eca2b1b-8e53-46d6-8476-fa00b8cfaabe None None] PATH_INFO: `/` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:34.657155 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 104/211] 10.222.0.27 () {54 vars in 927 bytes} [Tue Aug 30 01:42:34 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 01:42:34.662787 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e66d2fd6-e2e4-4b91-bd17-4aed4dc5eb85 None None] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:34.663075 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e66d2fd6-e2e4-4b91-bd17-4aed4dc5eb85 None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:34.663347 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e66d2fd6-e2e4-4b91-bd17-4aed4dc5eb85 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:34.684816 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.auth.core [None req-e66d2fd6-e2e4-4b91-bd17-4aed4dc5eb85 None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81578) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 01:42:34.686035 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-e66d2fd6-e2e4-4b91-bd17-4aed4dc5eb85 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:42:34.756332 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 108/212] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Aug 30 01:42:34 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 30 01:42:34.803796 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffd28ce6-6fc9-4b24-a9d0-a3036ecbc108 None None] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:34.804419 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffd28ce6-6fc9-4b24-a9d0-a3036ecbc108 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:34.804976 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffd28ce6-6fc9-4b24-a9d0-a3036ecbc108 None None] PATH_INFO: `/` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:34.807529 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 105/213] 10.222.0.27 () {54 vars in 927 bytes} [Tue Aug 30 01:42:34 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 01:42:34.816462 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-899dd72a-843f-41a4-a47e-56ac2828211b None None] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:34.817002 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-899dd72a-843f-41a4-a47e-56ac2828211b None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:34.817518 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-899dd72a-843f-41a4-a47e-56ac2828211b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:34.836801 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.auth.core [None req-899dd72a-843f-41a4-a47e-56ac2828211b None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81578) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 01:42:34.838604 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-899dd72a-843f-41a4-a47e-56ac2828211b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:42:34.862527 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:42:34.909172 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 109/214] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Aug 30 01:42:34 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Aug 30 01:42:34.937843 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffd28ce6-6fc9-4b24-a9d0-a3036ecbc108 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:42:35.002253 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:42:35.050449 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffd28ce6-6fc9-4b24-a9d0-a3036ecbc108 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:42:35.064819 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-899dd72a-843f-41a4-a47e-56ac2828211b None None] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:42:35.088013 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c936a28a-eda5-4fc9-9f52-b75b191c6983 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:42:35.088673 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c936a28a-eda5-4fc9-9f52-b75b191c6983 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:35.088931 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c936a28a-eda5-4fc9-9f52-b75b191c6983 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:35.089203 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c936a28a-eda5-4fc9-9f52-b75b191c6983 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:35.090351 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c936a28a-eda5-4fc9-9f52-b75b191c6983 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:42:35.091816 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c936a28a-eda5-4fc9-9f52-b75b191c6983 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:42:35.097922 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 106/215] 10.222.0.27 () {56 vars in 1108 bytes} [Tue Aug 30 01:42:34 2022] GET /identity/v3/domains/default => generated 199 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 30 01:42:35.099172 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c936a28a-eda5-4fc9-9f52-b75b191c6983 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:42:35.177022 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-899dd72a-843f-41a4-a47e-56ac2828211b None None] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:42:35.208034 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c936a28a-eda5-4fc9-9f52-b75b191c6983 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:42:35.210879 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-877b6f81-ef05-480e-bc6d-19ebabcc3065 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:42:35.211511 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-877b6f81-ef05-480e-bc6d-19ebabcc3065 None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:35.211789 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-877b6f81-ef05-480e-bc6d-19ebabcc3065 None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:35.212066 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-877b6f81-ef05-480e-bc6d-19ebabcc3065 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:35.222200 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-877b6f81-ef05-480e-bc6d-19ebabcc3065 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 30 01:42:35.227873 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application [None req-877b6f81-ef05-480e-bc6d-19ebabcc3065 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Aug 30 01:42:35.227873 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 01:42:35.227873 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 01:42:35.227873 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 01:42:35.227873 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 01:42:35.227873 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 01:42:35.227873 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 01:42:35.227873 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 01:42:35.227873 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 01:42:35.227873 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 01:42:35.227873 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 01:42:35.227873 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 01:42:35.227873 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 30 01:42:35.227873 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 30 01:42:35.227873 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 30 01:42:35.227873 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 30 01:42:35.228920 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 30 01:42:35.228920 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 30 01:42:35.228920 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 30 01:42:35.228920 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 30 01:42:35.228920 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 01:42:35.228920 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 01:42:35.228920 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 01:42:35.228920 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 01:42:35.228920 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 01:42:35.228920 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 01:42:35.228920 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 01:42:35.228920 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application with Lock( Aug 30 01:42:35.228920 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 01:42:35.228920 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._enter() Aug 30 01:42:35.228920 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 01:42:35.228920 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 01:42:35.229823 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 01:42:35.229823 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.creator() Aug 30 01:42:35.229823 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 01:42:35.229823 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application created_value = creator( Aug 30 01:42:35.229823 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 30 01:42:35.229823 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 30 01:42:35.229823 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 30 01:42:35.229823 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 30 01:42:35.229823 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 30 01:42:35.229823 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 30 01:42:35.229823 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Aug 30 01:42:35.229823 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application Aug 30 01:42:35.229823 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 110/216] 10.222.0.27 () {56 vars in 1099 bytes} [Tue Aug 30 01:42:35 2022] GET /identity/v3/roles/member => generated 84 bytes in 165 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 01:42:35.231211 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eadd4a62-bddb-4197-9bb9-f2c21ef4f1cd None None] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:35.231211 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eadd4a62-bddb-4197-9bb9-f2c21ef4f1cd None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:35.231369 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eadd4a62-bddb-4197-9bb9-f2c21ef4f1cd None None] PATH_INFO: `/` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:35.236489 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 111/217] 10.222.0.27 () {54 vars in 927 bytes} [Tue Aug 30 01:42:35 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 01:42:35.236489 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eadd4a62-bddb-4197-9bb9-f2c21ef4f1cd None None] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:42:35.238286 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:42:35.250318 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a7a4f24-9855-44b4-b9b9-ed22c477ebf6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:42:35.250927 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a7a4f24-9855-44b4-b9b9-ed22c477ebf6 None admin] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:35.255546 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a7a4f24-9855-44b4-b9b9-ed22c477ebf6 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:35.255546 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a7a4f24-9855-44b4-b9b9-ed22c477ebf6 None admin] PATH_INFO: `/v3/users` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:35.255546 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a7a4f24-9855-44b4-b9b9-ed22c477ebf6 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:42:35.255932 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:42:35.256041 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a7a4f24-9855-44b4-b9b9-ed22c477ebf6 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:42:35.337511 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eadd4a62-bddb-4197-9bb9-f2c21ef4f1cd None None] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:42:35.345166 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 107/218] 10.222.0.27 () {60 vars in 1132 bytes} [Tue Aug 30 01:42:35 2022] POST /identity/v3/users => generated 297 bytes in 246 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 30 01:42:35.346103 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a7a4f24-9855-44b4-b9b9-ed22c477ebf6 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:42:35.393172 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3359e1b-c66c-426f-9de8-75038210e36c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:42:35.393856 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3359e1b-c66c-426f-9de8-75038210e36c None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:35.394138 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3359e1b-c66c-426f-9de8-75038210e36c None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:35.394408 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3359e1b-c66c-426f-9de8-75038210e36c None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:35.406652 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3359e1b-c66c-426f-9de8-75038210e36c None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 30 01:42:35.411709 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application [None req-d3359e1b-c66c-426f-9de8-75038210e36c None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 30 01:42:35.411709 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 01:42:35.411709 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 01:42:35.411709 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 01:42:35.411709 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 01:42:35.411709 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 01:42:35.411709 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 01:42:35.411709 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 01:42:35.411709 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 01:42:35.411709 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 01:42:35.411709 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 01:42:35.411709 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 01:42:35.411709 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 30 01:42:35.411709 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 30 01:42:35.411709 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 30 01:42:35.411709 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 30 01:42:35.412628 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 30 01:42:35.412628 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 30 01:42:35.412628 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 30 01:42:35.412628 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 30 01:42:35.412628 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 01:42:35.412628 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 01:42:35.412628 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 01:42:35.412628 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 01:42:35.412628 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 01:42:35.412628 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 01:42:35.412628 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 01:42:35.412628 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application with Lock( Aug 30 01:42:35.412628 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 01:42:35.412628 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._enter() Aug 30 01:42:35.412628 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 01:42:35.412628 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 01:42:35.413878 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 01:42:35.413878 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.creator() Aug 30 01:42:35.413878 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 01:42:35.413878 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application created_value = creator( Aug 30 01:42:35.413878 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 30 01:42:35.413878 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 30 01:42:35.413878 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 30 01:42:35.413878 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 30 01:42:35.413878 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 30 01:42:35.413878 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 30 01:42:35.413878 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 30 01:42:35.413878 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application Aug 30 01:42:35.413878 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 112/219] 10.222.0.27 () {56 vars in 1096 bytes} [Tue Aug 30 01:42:35 2022] GET /identity/v3/roles/admin => generated 83 bytes in 180 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 01:42:35.414702 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5611ae37-421e-465a-849a-79e231529d4e None None] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:35.414702 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5611ae37-421e-465a-849a-79e231529d4e None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:35.414702 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5611ae37-421e-465a-849a-79e231529d4e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:35.435944 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.auth.core [None req-5611ae37-421e-465a-849a-79e231529d4e None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81578) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 01:42:35.544318 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-5611ae37-421e-465a-849a-79e231529d4e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:42:35.544458 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a7a4f24-9855-44b4-b9b9-ed22c477ebf6 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:42:35.544549 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 113/220] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Aug 30 01:42:35 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 111 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 30 01:42:35.544549 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5611ae37-421e-465a-849a-79e231529d4e None None] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:42:35.544696 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e3f0988-83c2-44c5-80ba-c3e44a7bb377 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:42:35.544696 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e3f0988-83c2-44c5-80ba-c3e44a7bb377 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:35.544696 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e3f0988-83c2-44c5-80ba-c3e44a7bb377 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:35.544696 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e3f0988-83c2-44c5-80ba-c3e44a7bb377 None admin] PATH_INFO: `/v3/roles` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:35.544968 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e3f0988-83c2-44c5-80ba-c3e44a7bb377 None admin] RBAC: Adding query filter params (name=member) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:42:35.544968 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e3f0988-83c2-44c5-80ba-c3e44a7bb377 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:42:35.544968 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e3f0988-83c2-44c5-80ba-c3e44a7bb377 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:42:35.545155 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:42:35.545230 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 108/221] 10.222.0.27 () {56 vars in 1101 bytes} [Tue Aug 30 01:42:35 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 197 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:42:35.548106 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e3f0988-83c2-44c5-80ba-c3e44a7bb377 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:42:35.612714 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:42:35.695032 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5611ae37-421e-465a-849a-79e231529d4e None None] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:42:35.709025 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e3f0988-83c2-44c5-80ba-c3e44a7bb377 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:42:35.787808 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7ea100e-043c-4801-b859-13f9b4c05c21 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:42:35.788580 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7ea100e-043c-4801-b859-13f9b4c05c21 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:35.789038 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7ea100e-043c-4801-b859-13f9b4c05c21 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:35.789373 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7ea100e-043c-4801-b859-13f9b4c05c21 None admin] PATH_INFO: `/v3/users/b3189d44e5c846e88be8df57633d93cd` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:35.791452 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45924ac5-263a-4519-86a4-cc2a04ef37f8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:42:35.792344 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45924ac5-263a-4519-86a4-cc2a04ef37f8 None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:35.792780 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45924ac5-263a-4519-86a4-cc2a04ef37f8 None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:35.793191 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45924ac5-263a-4519-86a4-cc2a04ef37f8 None admin] PATH_INFO: `/v3/roles` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:35.794050 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45924ac5-263a-4519-86a4-cc2a04ef37f8 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81578) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:42:35.794670 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45924ac5-263a-4519-86a4-cc2a04ef37f8 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:42:35.797324 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45924ac5-263a-4519-86a4-cc2a04ef37f8 None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:42:35.806296 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7ea100e-043c-4801-b859-13f9b4c05c21 None admin] RBAC: Authorizing `identity:get_user(user_id=b3189d44e5c846e88be8df57633d93cd)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:42:35.808130 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7ea100e-043c-4801-b859-13f9b4c05c21 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:42:35.809971 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 114/222] 10.222.0.27 () {56 vars in 1099 bytes} [Tue Aug 30 01:42:35 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 285 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:42:35.810104 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 109/223] 10.222.0.27 () {56 vars in 1177 bytes} [Tue Aug 30 01:42:35 2022] GET /identity/v3/users/b3189d44e5c846e88be8df57633d93cd => generated 297 bytes in 262 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 30 01:42:35.816594 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7ea100e-043c-4801-b859-13f9b4c05c21 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:42:35.818030 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45924ac5-263a-4519-86a4-cc2a04ef37f8 None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:42:35.907552 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:42:35.933911 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7ea100e-043c-4801-b859-13f9b4c05c21 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:42:35.943737 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45924ac5-263a-4519-86a4-cc2a04ef37f8 None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:42:35.947308 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:42:35.981617 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5150247b-68e2-44f2-801e-8f51e4a88b41 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:42:35.982554 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5150247b-68e2-44f2-801e-8f51e4a88b41 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:35.983016 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5150247b-68e2-44f2-801e-8f51e4a88b41 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:35.983564 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5150247b-68e2-44f2-801e-8f51e4a88b41 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:35.990566 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2cab31d-9c16-4deb-804a-d75c689d487f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:42:35.991217 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2cab31d-9c16-4deb-804a-d75c689d487f None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:35.991497 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2cab31d-9c16-4deb-804a-d75c689d487f None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:35.991769 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2cab31d-9c16-4deb-804a-d75c689d487f None admin] PATH_INFO: `/v3/users/a4a8afdab07d4fd089132965087a3d56` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:35.993435 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2cab31d-9c16-4deb-804a-d75c689d487f None admin] RBAC: Authorizing `identity:get_user(user_id=a4a8afdab07d4fd089132965087a3d56)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:42:35.994977 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2cab31d-9c16-4deb-804a-d75c689d487f None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:42:35.996915 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 115/224] 10.222.0.27 () {56 vars in 1177 bytes} [Tue Aug 30 01:42:35 2022] GET /identity/v3/users/a4a8afdab07d4fd089132965087a3d56 => generated 283 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:42:35.997529 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2cab31d-9c16-4deb-804a-d75c689d487f None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:42:36.063985 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5150247b-68e2-44f2-801e-8f51e4a88b41 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:42:36.066090 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5150247b-68e2-44f2-801e-8f51e4a88b41 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:42:36.122185 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2cab31d-9c16-4deb-804a-d75c689d487f None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:42:36.161317 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-881c0f99-a840-4602-86b7-c3ae02c0d49e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:42:36.161959 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-881c0f99-a840-4602-86b7-c3ae02c0d49e None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:36.162243 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-881c0f99-a840-4602-86b7-c3ae02c0d49e None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:36.162518 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-881c0f99-a840-4602-86b7-c3ae02c0d49e None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:36.173273 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:42:36.177082 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-881c0f99-a840-4602-86b7-c3ae02c0d49e None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 30 01:42:36.181920 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 110/225] 10.222.0.27 () {58 vars in 1282 bytes} [Tue Aug 30 01:42:35 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 366 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Aug 30 01:42:36.191549 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5150247b-68e2-44f2-801e-8f51e4a88b41 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:42:36.191682 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application [None req-881c0f99-a840-4602-86b7-c3ae02c0d49e None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Aug 30 01:42:36.191682 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 01:42:36.191682 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 01:42:36.191682 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 01:42:36.191682 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 01:42:36.191682 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 01:42:36.191682 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 01:42:36.191682 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 01:42:36.191682 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 01:42:36.191682 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 01:42:36.191682 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 01:42:36.191682 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 01:42:36.191682 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 30 01:42:36.191682 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 30 01:42:36.191682 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 30 01:42:36.191682 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 30 01:42:36.192559 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 30 01:42:36.192559 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 30 01:42:36.192559 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 30 01:42:36.192559 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 30 01:42:36.192559 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 01:42:36.192559 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 01:42:36.192559 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 01:42:36.192559 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 01:42:36.192559 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 01:42:36.192559 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 01:42:36.192559 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 01:42:36.192559 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application with Lock( Aug 30 01:42:36.192559 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 01:42:36.192559 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._enter() Aug 30 01:42:36.192559 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 01:42:36.192559 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 01:42:36.193374 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 01:42:36.193374 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.creator() Aug 30 01:42:36.193374 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 01:42:36.193374 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application created_value = creator( Aug 30 01:42:36.193374 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 30 01:42:36.193374 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 30 01:42:36.193374 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 30 01:42:36.193374 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 30 01:42:36.193374 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 30 01:42:36.193374 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 30 01:42:36.193374 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Aug 30 01:42:36.193374 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application Aug 30 01:42:36.193374 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 116/226] 10.222.0.27 () {56 vars in 1114 bytes} [Tue Aug 30 01:42:35 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 193 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 01:42:36.193374 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-881c0f99-a840-4602-86b7-c3ae02c0d49e None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:42:36.196855 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:42:36.331517 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5150247b-68e2-44f2-801e-8f51e4a88b41 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:42:36.336523 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-881c0f99-a840-4602-86b7-c3ae02c0d49e None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:42:36.385435 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18efd661-09b1-45b4-a77f-797c7b9060b5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:42:36.386089 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18efd661-09b1-45b4-a77f-797c7b9060b5 None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:36.386409 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18efd661-09b1-45b4-a77f-797c7b9060b5 None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:36.386888 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18efd661-09b1-45b4-a77f-797c7b9060b5 None admin] PATH_INFO: `/v3/projects/67913d5682a443069a5eed96f179751f` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:36.388735 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18efd661-09b1-45b4-a77f-797c7b9060b5 None admin] RBAC: Authorizing `identity:get_project(project_id=67913d5682a443069a5eed96f179751f)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:42:36.390421 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18efd661-09b1-45b4-a77f-797c7b9060b5 None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:42:36.391975 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6c8059e-b769-40d5-8ee5-d0d08d37c1d6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:42:36.393057 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 117/227] 10.222.0.27 () {56 vars in 1186 bytes} [Tue Aug 30 01:42:36 2022] GET /identity/v3/projects/67913d5682a443069a5eed96f179751f => generated 302 bytes in 202 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 30 01:42:36.393692 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6c8059e-b769-40d5-8ee5-d0d08d37c1d6 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:36.394448 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6c8059e-b769-40d5-8ee5-d0d08d37c1d6 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:36.395267 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6c8059e-b769-40d5-8ee5-d0d08d37c1d6 None admin] PATH_INFO: `/v3/projects/67913d5682a443069a5eed96f179751f` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:36.396930 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6c8059e-b769-40d5-8ee5-d0d08d37c1d6 None admin] RBAC: Authorizing `identity:get_project(project_id=67913d5682a443069a5eed96f179751f)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:42:36.398703 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6c8059e-b769-40d5-8ee5-d0d08d37c1d6 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:42:36.399250 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18efd661-09b1-45b4-a77f-797c7b9060b5 None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:42:36.401346 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 111/228] 10.222.0.27 () {56 vars in 1186 bytes} [Tue Aug 30 01:42:36 2022] GET /identity/v3/projects/67913d5682a443069a5eed96f179751f => generated 302 bytes in 219 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:42:36.403482 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6c8059e-b769-40d5-8ee5-d0d08d37c1d6 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:42:36.495022 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:42:36.502176 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:42:36.541272 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18efd661-09b1-45b4-a77f-797c7b9060b5 None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:42:36.545636 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6c8059e-b769-40d5-8ee5-d0d08d37c1d6 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:42:36.586025 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57b47055-e98e-4d58-a5d6-f5336371e64a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:42:36.587591 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57b47055-e98e-4d58-a5d6-f5336371e64a None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:36.588078 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57b47055-e98e-4d58-a5d6-f5336371e64a None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:36.588615 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57b47055-e98e-4d58-a5d6-f5336371e64a None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:36.589803 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57b47055-e98e-4d58-a5d6-f5336371e64a None admin] RBAC: Adding query filter params (role.id=1fb81afcb9f245f3b989add94598a063, scope.project.id=67913d5682a443069a5eed96f179751f, user.id=b3189d44e5c846e88be8df57633d93cd) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:42:36.590327 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57b47055-e98e-4d58-a5d6-f5336371e64a None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:42:36.592231 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57b47055-e98e-4d58-a5d6-f5336371e64a None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:42:36.596073 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e952140c-8567-4ff3-b2df-7a2ecdd54b17 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:42:36.596852 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e952140c-8567-4ff3-b2df-7a2ecdd54b17 None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:36.597331 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e952140c-8567-4ff3-b2df-7a2ecdd54b17 None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:36.597832 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e952140c-8567-4ff3-b2df-7a2ecdd54b17 None admin] PATH_INFO: `/v3/roles` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:36.599002 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e952140c-8567-4ff3-b2df-7a2ecdd54b17 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81578) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:42:36.599497 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e952140c-8567-4ff3-b2df-7a2ecdd54b17 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:42:36.600269 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 112/229] 10.222.0.27 () {56 vars in 1374 bytes} [Tue Aug 30 01:42:36 2022] GET /identity/v3/role_assignments?user.id=b3189d44e5c846e88be8df57633d93cd&scope.project.id=67913d5682a443069a5eed96f179751f&role.id=1fb81afcb9f245f3b989add94598a063 => generated 259 bytes in 198 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:42:36.601689 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57b47055-e98e-4d58-a5d6-f5336371e64a None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:42:36.606439 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e952140c-8567-4ff3-b2df-7a2ecdd54b17 None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:42:36.626052 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 118/230] 10.222.0.27 () {56 vars in 1111 bytes} [Tue Aug 30 01:42:36 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 232 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:42:36.632770 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e952140c-8567-4ff3-b2df-7a2ecdd54b17 None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:42:36.720688 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57b47055-e98e-4d58-a5d6-f5336371e64a None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:42:36.759177 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:42:36.765166 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c91d27a8-a02a-40f5-9571-e2e94c15c21b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:42:36.766258 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c91d27a8-a02a-40f5-9571-e2e94c15c21b None admin] REQUEST_METHOD: `PUT` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:36.766745 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c91d27a8-a02a-40f5-9571-e2e94c15c21b None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:36.767207 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c91d27a8-a02a-40f5-9571-e2e94c15c21b None admin] PATH_INFO: `/v3/projects/67913d5682a443069a5eed96f179751f/users/a4a8afdab07d4fd089132965087a3d56/roles/ead5fac86a0747ea8823a82efcea8ed7` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:36.769850 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c91d27a8-a02a-40f5-9571-e2e94c15c21b None admin] RBAC: Authorizing `identity:create_grant(project_id=67913d5682a443069a5eed96f179751f, user_id=a4a8afdab07d4fd089132965087a3d56, role_id=ead5fac86a0747ea8823a82efcea8ed7)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:42:36.775763 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c91d27a8-a02a-40f5-9571-e2e94c15c21b None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:42:36.816715 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e952140c-8567-4ff3-b2df-7a2ecdd54b17 None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:42:36.849471 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:42:36.853744 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 113/231] 10.222.0.27 () {58 vars in 1439 bytes} [Tue Aug 30 01:42:36 2022] PUT /identity/v3/projects/67913d5682a443069a5eed96f179751f/users/a4a8afdab07d4fd089132965087a3d56/roles/ead5fac86a0747ea8823a82efcea8ed7 => generated 0 bytes in 253 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 01:42:36.867374 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da106471-3b4c-46d7-b497-247249a5c12d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:42:36.868829 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da106471-3b4c-46d7-b497-247249a5c12d None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:36.869218 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da106471-3b4c-46d7-b497-247249a5c12d None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:36.869513 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da106471-3b4c-46d7-b497-247249a5c12d None admin] PATH_INFO: `/v3/users/a4a8afdab07d4fd089132965087a3d56` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:36.870786 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da106471-3b4c-46d7-b497-247249a5c12d None admin] RBAC: Authorizing `identity:get_user(user_id=a4a8afdab07d4fd089132965087a3d56)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:42:36.872414 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da106471-3b4c-46d7-b497-247249a5c12d None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:42:36.874271 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 119/232] 10.222.0.27 () {56 vars in 1177 bytes} [Tue Aug 30 01:42:36 2022] GET /identity/v3/users/a4a8afdab07d4fd089132965087a3d56 => generated 283 bytes in 243 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:42:36.880343 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c91d27a8-a02a-40f5-9571-e2e94c15c21b None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:42:37.002573 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c91d27a8-a02a-40f5-9571-e2e94c15c21b None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:42:37.040575 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2278ec06-cbad-44f8-a75c-0ca65187e7f1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:42:37.041584 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2278ec06-cbad-44f8-a75c-0ca65187e7f1 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:37.042057 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2278ec06-cbad-44f8-a75c-0ca65187e7f1 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:37.042504 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2278ec06-cbad-44f8-a75c-0ca65187e7f1 None admin] PATH_INFO: `/v3/projects/67913d5682a443069a5eed96f179751f` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:37.043977 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2278ec06-cbad-44f8-a75c-0ca65187e7f1 None admin] RBAC: Authorizing `identity:get_project(project_id=67913d5682a443069a5eed96f179751f)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:42:37.045667 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2278ec06-cbad-44f8-a75c-0ca65187e7f1 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:42:37.047529 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 114/233] 10.222.0.27 () {56 vars in 1186 bytes} [Tue Aug 30 01:42:36 2022] GET /identity/v3/projects/67913d5682a443069a5eed96f179751f => generated 302 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:42:37.053266 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da106471-3b4c-46d7-b497-247249a5c12d None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:42:37.177946 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da106471-3b4c-46d7-b497-247249a5c12d None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:42:37.219544 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ff227cf-0fe8-4cbe-88fc-90efa77debdc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:42:37.220517 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ff227cf-0fe8-4cbe-88fc-90efa77debdc None admin] REQUEST_METHOD: `PUT` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:37.220997 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ff227cf-0fe8-4cbe-88fc-90efa77debdc None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:37.221532 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ff227cf-0fe8-4cbe-88fc-90efa77debdc None admin] PATH_INFO: `/v3/projects/67913d5682a443069a5eed96f179751f/users/a4a8afdab07d4fd089132965087a3d56/roles/4621068972ba4f79bad104f956db04aa` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:37.224009 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ff227cf-0fe8-4cbe-88fc-90efa77debdc None admin] RBAC: Authorizing `identity:create_grant(project_id=67913d5682a443069a5eed96f179751f, user_id=a4a8afdab07d4fd089132965087a3d56, role_id=4621068972ba4f79bad104f956db04aa)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:42:37.225645 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ff227cf-0fe8-4cbe-88fc-90efa77debdc None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:42:37.352386 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 120/234] 10.222.0.27 () {58 vars in 1439 bytes} [Tue Aug 30 01:42:37 2022] PUT /identity/v3/projects/67913d5682a443069a5eed96f179751f/users/a4a8afdab07d4fd089132965087a3d56/roles/4621068972ba4f79bad104f956db04aa => generated 0 bytes in 299 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 30 01:42:37.945658 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d719ef43-a69e-4904-9d79-583d48755c8b None None] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:37.947382 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d719ef43-a69e-4904-9d79-583d48755c8b None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:37.947867 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d719ef43-a69e-4904-9d79-583d48755c8b None None] PATH_INFO: `/` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:37.949801 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 115/235] 10.222.0.27 () {54 vars in 927 bytes} [Tue Aug 30 01:42:37 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 01:42:37.954928 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f59a2ee-4f42-4c7f-8902-1a30a6a0bad3 None None] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:37.955212 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f59a2ee-4f42-4c7f-8902-1a30a6a0bad3 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:37.955483 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f59a2ee-4f42-4c7f-8902-1a30a6a0bad3 None None] PATH_INFO: `/` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:37.956918 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 116/236] 10.222.0.27 () {54 vars in 927 bytes} [Tue Aug 30 01:42:37 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 01:42:37.958348 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11e5ae80-ed42-42f0-970f-5db0f6e0b7f5 None None] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:37.958635 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11e5ae80-ed42-42f0-970f-5db0f6e0b7f5 None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:37.958907 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11e5ae80-ed42-42f0-970f-5db0f6e0b7f5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:37.966392 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fff1edc-4f46-462a-86b6-eabb02637d77 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:37.966392 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fff1edc-4f46-462a-86b6-eabb02637d77 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:37.966392 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fff1edc-4f46-462a-86b6-eabb02637d77 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:37.991013 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.auth.core [None req-11e5ae80-ed42-42f0-970f-5db0f6e0b7f5 None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81578) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 01:42:37.992223 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-4fff1edc-4f46-462a-86b6-eabb02637d77 None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 01:42:37.993735 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-4fff1edc-4f46-462a-86b6-eabb02637d77 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:42:37.995131 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-11e5ae80-ed42-42f0-970f-5db0f6e0b7f5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:42:38.080654 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:42:38.093874 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 117/237] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Aug 30 01:42:37 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 132 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Aug 30 01:42:38.123191 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:42:38.249691 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 121/238] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Aug 30 01:42:37 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 194 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 30 01:42:38.488105 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fff1edc-4f46-462a-86b6-eabb02637d77 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:42:38.527529 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11e5ae80-ed42-42f0-970f-5db0f6e0b7f5 None None] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:42:38.650471 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fff1edc-4f46-462a-86b6-eabb02637d77 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:42:38.716124 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d55c34c0-7e06-45b2-85c8-c20bbe0b3a95 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:42:38.719324 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d55c34c0-7e06-45b2-85c8-c20bbe0b3a95 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:38.719324 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d55c34c0-7e06-45b2-85c8-c20bbe0b3a95 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:38.719324 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d55c34c0-7e06-45b2-85c8-c20bbe0b3a95 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:38.721157 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d55c34c0-7e06-45b2-85c8-c20bbe0b3a95 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:42:38.729481 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d55c34c0-7e06-45b2-85c8-c20bbe0b3a95 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:42:38.729650 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11e5ae80-ed42-42f0-970f-5db0f6e0b7f5 None None] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:42:38.729943 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 118/239] 10.222.0.27 () {56 vars in 1108 bytes} [Tue Aug 30 01:42:38 2022] GET /identity/v3/domains/default => generated 199 bytes in 242 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:42:38.731215 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d55c34c0-7e06-45b2-85c8-c20bbe0b3a95 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:42:38.799953 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-645352a1-fb27-4c91-ab02-e57c549a7d86 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:42:38.800591 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-645352a1-fb27-4c91-ab02-e57c549a7d86 None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:38.800865 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-645352a1-fb27-4c91-ab02-e57c549a7d86 None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:38.801137 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-645352a1-fb27-4c91-ab02-e57c549a7d86 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:38.827533 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-645352a1-fb27-4c91-ab02-e57c549a7d86 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 30 01:42:38.833273 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application [None req-645352a1-fb27-4c91-ab02-e57c549a7d86 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Aug 30 01:42:38.833273 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 01:42:38.833273 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 01:42:38.833273 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 01:42:38.833273 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 01:42:38.833273 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 01:42:38.833273 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 01:42:38.833273 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 01:42:38.833273 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 01:42:38.833273 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 01:42:38.833273 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 01:42:38.833273 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 01:42:38.833273 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 30 01:42:38.833273 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 30 01:42:38.833273 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 30 01:42:38.833273 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 30 01:42:38.834227 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 30 01:42:38.834227 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 30 01:42:38.834227 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 30 01:42:38.834227 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 30 01:42:38.834227 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 01:42:38.834227 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 01:42:38.834227 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 01:42:38.834227 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 01:42:38.834227 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 01:42:38.834227 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 01:42:38.834227 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 01:42:38.834227 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application with Lock( Aug 30 01:42:38.834227 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 01:42:38.834227 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._enter() Aug 30 01:42:38.834227 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 01:42:38.834227 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 01:42:38.835027 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 01:42:38.835027 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.creator() Aug 30 01:42:38.835027 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 01:42:38.835027 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application created_value = creator( Aug 30 01:42:38.835027 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 30 01:42:38.835027 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 30 01:42:38.835027 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 30 01:42:38.835027 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 30 01:42:38.835027 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 30 01:42:38.835027 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 30 01:42:38.835027 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Aug 30 01:42:38.835027 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application Aug 30 01:42:38.845280 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 122/240] 10.222.0.27 () {56 vars in 1099 bytes} [Tue Aug 30 01:42:38 2022] GET /identity/v3/roles/reader => generated 84 bytes in 318 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 01:42:38.846325 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-645352a1-fb27-4c91-ab02-e57c549a7d86 None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:42:38.861822 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:42:38.935834 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:42:38.951646 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d55c34c0-7e06-45b2-85c8-c20bbe0b3a95 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:42:39.004710 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-645352a1-fb27-4c91-ab02-e57c549a7d86 None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:42:39.028810 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-508b3298-4563-49cf-a86a-f297456b62bd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:42:39.029427 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-508b3298-4563-49cf-a86a-f297456b62bd None admin] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:39.029727 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-508b3298-4563-49cf-a86a-f297456b62bd None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:39.030741 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-508b3298-4563-49cf-a86a-f297456b62bd None admin] PATH_INFO: `/v3/users` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:39.031429 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-508b3298-4563-49cf-a86a-f297456b62bd None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:42:39.032902 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-508b3298-4563-49cf-a86a-f297456b62bd None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:42:39.065927 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5017baff-7ded-4535-9255-76395dac6ee2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:42:39.065927 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5017baff-7ded-4535-9255-76395dac6ee2 None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:39.066172 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5017baff-7ded-4535-9255-76395dac6ee2 None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:39.066496 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5017baff-7ded-4535-9255-76395dac6ee2 None admin] PATH_INFO: `/v3/roles` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:39.067174 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5017baff-7ded-4535-9255-76395dac6ee2 None admin] RBAC: Adding query filter params (name=reader) {{(pid=81578) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:42:39.067473 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5017baff-7ded-4535-9255-76395dac6ee2 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:42:39.068878 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5017baff-7ded-4535-9255-76395dac6ee2 None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:42:39.079308 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 123/241] 10.222.0.27 () {56 vars in 1101 bytes} [Tue Aug 30 01:42:38 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 233 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:42:39.080821 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d0dd9c6-26c5-4d70-96fe-fc63731063c1 None None] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:39.081107 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d0dd9c6-26c5-4d70-96fe-fc63731063c1 None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:39.081377 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d0dd9c6-26c5-4d70-96fe-fc63731063c1 None None] PATH_INFO: `/` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:39.082658 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 124/242] 10.222.0.27 () {54 vars in 927 bytes} [Tue Aug 30 01:42:39 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 01:42:39.085749 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d0dd9c6-26c5-4d70-96fe-fc63731063c1 None None] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:42:39.121600 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 119/243] 10.222.0.27 () {60 vars in 1132 bytes} [Tue Aug 30 01:42:38 2022] POST /identity/v3/users => generated 293 bytes in 391 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 01:42:39.122958 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a1fe575-a926-4c93-af25-e012b7e9b9f4 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:39.123241 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a1fe575-a926-4c93-af25-e012b7e9b9f4 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:39.123559 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a1fe575-a926-4c93-af25-e012b7e9b9f4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:39.151509 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-6a1fe575-a926-4c93-af25-e012b7e9b9f4 None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 01:42:39.153039 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-6a1fe575-a926-4c93-af25-e012b7e9b9f4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:42:39.254945 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d0dd9c6-26c5-4d70-96fe-fc63731063c1 None None] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:42:39.266378 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:42:39.278071 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 120/244] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Aug 30 01:42:39 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 149 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 30 01:42:39.278071 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39705785-19d8-4710-ab62-b5c9c7805f70 None None] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:39.278071 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39705785-19d8-4710-ab62-b5c9c7805f70 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:39.278071 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39705785-19d8-4710-ab62-b5c9c7805f70 None None] PATH_INFO: `/` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:39.282771 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 121/245] 10.222.0.27 () {54 vars in 927 bytes} [Tue Aug 30 01:42:39 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 01:42:39.287245 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a205a52-0e9c-4a7f-aff9-7a2655c930b9 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:39.287536 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a205a52-0e9c-4a7f-aff9-7a2655c930b9 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:39.290710 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a205a52-0e9c-4a7f-aff9-7a2655c930b9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:39.320014 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83a9f080-49a1-4f88-9928-44d01f9bb3d6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:42:39.320634 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83a9f080-49a1-4f88-9928-44d01f9bb3d6 None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:39.321559 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83a9f080-49a1-4f88-9928-44d01f9bb3d6 None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:39.322062 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83a9f080-49a1-4f88-9928-44d01f9bb3d6 None admin] PATH_INFO: `/v3/users/2ed4eca98551441e8a808afce7833f89` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:39.323289 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-3a205a52-0e9c-4a7f-aff9-7a2655c930b9 None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 01:42:39.324524 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-3a205a52-0e9c-4a7f-aff9-7a2655c930b9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:42:39.342062 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83a9f080-49a1-4f88-9928-44d01f9bb3d6 None admin] RBAC: Authorizing `identity:get_user(user_id=2ed4eca98551441e8a808afce7833f89)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:42:39.343711 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83a9f080-49a1-4f88-9928-44d01f9bb3d6 None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:42:39.346229 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 125/246] 10.222.0.27 () {56 vars in 1177 bytes} [Tue Aug 30 01:42:39 2022] GET /identity/v3/users/2ed4eca98551441e8a808afce7833f89 => generated 297 bytes in 261 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 30 01:42:39.352080 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83a9f080-49a1-4f88-9928-44d01f9bb3d6 None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:42:39.380587 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:42:39.414852 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 122/247] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Aug 30 01:42:39 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 131 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 30 01:42:39.484046 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83a9f080-49a1-4f88-9928-44d01f9bb3d6 None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:42:39.523800 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7077cb77-3c6a-4577-90ea-5fb3dce1b92c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:42:39.525198 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7077cb77-3c6a-4577-90ea-5fb3dce1b92c None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:39.525498 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7077cb77-3c6a-4577-90ea-5fb3dce1b92c None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:39.525775 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7077cb77-3c6a-4577-90ea-5fb3dce1b92c None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:39.567426 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:42:39.606005 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a205a52-0e9c-4a7f-aff9-7a2655c930b9 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:42:39.615230 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7077cb77-3c6a-4577-90ea-5fb3dce1b92c None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:42:39.616828 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7077cb77-3c6a-4577-90ea-5fb3dce1b92c None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:42:39.747517 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a205a52-0e9c-4a7f-aff9-7a2655c930b9 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:42:39.755288 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 126/248] 10.222.0.27 () {58 vars in 1282 bytes} [Tue Aug 30 01:42:39 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 404 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 30 01:42:39.761555 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7077cb77-3c6a-4577-90ea-5fb3dce1b92c None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:42:39.795424 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2aeef0b-0cc5-49bf-8842-3aaf3fd6be6c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:42:39.796173 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2aeef0b-0cc5-49bf-8842-3aaf3fd6be6c None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:39.796458 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2aeef0b-0cc5-49bf-8842-3aaf3fd6be6c None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:39.796734 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2aeef0b-0cc5-49bf-8842-3aaf3fd6be6c None admin] PATH_INFO: `/v3/roles/member` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:39.818128 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2aeef0b-0cc5-49bf-8842-3aaf3fd6be6c None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 30 01:42:39.833497 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application [None req-f2aeef0b-0cc5-49bf-8842-3aaf3fd6be6c None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Aug 30 01:42:39.833497 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 01:42:39.833497 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 01:42:39.833497 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 01:42:39.833497 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 01:42:39.833497 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 01:42:39.833497 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 01:42:39.833497 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 01:42:39.833497 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 01:42:39.833497 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 01:42:39.833497 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 01:42:39.833497 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 01:42:39.833497 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 30 01:42:39.833497 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 30 01:42:39.833497 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 30 01:42:39.833497 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 30 01:42:39.834459 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 30 01:42:39.834459 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 30 01:42:39.834459 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 30 01:42:39.834459 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 30 01:42:39.834459 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 01:42:39.834459 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 01:42:39.834459 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 01:42:39.834459 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 01:42:39.834459 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 01:42:39.834459 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 01:42:39.834459 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 01:42:39.834459 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application with Lock( Aug 30 01:42:39.834459 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 01:42:39.834459 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._enter() Aug 30 01:42:39.834459 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 01:42:39.834459 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 01:42:39.835256 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 01:42:39.835256 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.creator() Aug 30 01:42:39.835256 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 01:42:39.835256 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application created_value = creator( Aug 30 01:42:39.835256 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 30 01:42:39.835256 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 30 01:42:39.835256 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 30 01:42:39.835256 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 30 01:42:39.835256 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 30 01:42:39.835256 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 30 01:42:39.835256 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Aug 30 01:42:39.835256 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Aug 30 01:42:39.835256 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 123/249] 10.222.0.27 () {56 vars in 1099 bytes} [Tue Aug 30 01:42:39 2022] GET /identity/v3/roles/member => generated 84 bytes in 230 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 01:42:39.835960 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2aeef0b-0cc5-49bf-8842-3aaf3fd6be6c None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:42:39.870975 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:42:39.898562 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:42:39.931961 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7077cb77-3c6a-4577-90ea-5fb3dce1b92c None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:42:39.972805 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9de9235f-7709-46b3-a7a7-8ae6062a253f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:42:39.973445 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9de9235f-7709-46b3-a7a7-8ae6062a253f None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:39.973890 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9de9235f-7709-46b3-a7a7-8ae6062a253f None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:39.974122 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9de9235f-7709-46b3-a7a7-8ae6062a253f None admin] PATH_INFO: `/v3/projects/67913d5682a443069a5eed96f179751f` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:39.975402 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9de9235f-7709-46b3-a7a7-8ae6062a253f None admin] RBAC: Authorizing `identity:get_project(project_id=67913d5682a443069a5eed96f179751f)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:42:39.977175 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9de9235f-7709-46b3-a7a7-8ae6062a253f None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:42:39.980244 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 127/250] 10.222.0.27 () {56 vars in 1186 bytes} [Tue Aug 30 01:42:39 2022] GET /identity/v3/projects/67913d5682a443069a5eed96f179751f => generated 302 bytes in 219 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 30 01:42:39.981083 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9de9235f-7709-46b3-a7a7-8ae6062a253f None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:42:39.993343 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2aeef0b-0cc5-49bf-8842-3aaf3fd6be6c None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:42:40.032330 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba0bf076-0236-48a0-a9cb-7ed1c5ab0523 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:42:40.033425 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba0bf076-0236-48a0-a9cb-7ed1c5ab0523 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:40.034105 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba0bf076-0236-48a0-a9cb-7ed1c5ab0523 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:40.034589 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba0bf076-0236-48a0-a9cb-7ed1c5ab0523 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:40.047251 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba0bf076-0236-48a0-a9cb-7ed1c5ab0523 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 30 01:42:40.052967 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application [None req-ba0bf076-0236-48a0-a9cb-7ed1c5ab0523 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 30 01:42:40.052967 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 01:42:40.052967 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 01:42:40.052967 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 01:42:40.052967 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 01:42:40.052967 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 01:42:40.052967 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 01:42:40.052967 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 01:42:40.052967 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 01:42:40.052967 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 01:42:40.052967 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 01:42:40.052967 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 01:42:40.052967 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 30 01:42:40.052967 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 30 01:42:40.052967 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 30 01:42:40.052967 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 30 01:42:40.053903 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 30 01:42:40.053903 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 30 01:42:40.053903 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 30 01:42:40.053903 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 30 01:42:40.053903 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 01:42:40.053903 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 01:42:40.053903 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 01:42:40.053903 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 01:42:40.053903 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 01:42:40.053903 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 01:42:40.053903 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 01:42:40.053903 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application with Lock( Aug 30 01:42:40.053903 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 01:42:40.053903 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._enter() Aug 30 01:42:40.053903 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 01:42:40.053903 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 01:42:40.054703 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 01:42:40.054703 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.creator() Aug 30 01:42:40.054703 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 01:42:40.054703 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application created_value = creator( Aug 30 01:42:40.054703 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 30 01:42:40.054703 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 30 01:42:40.054703 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 30 01:42:40.054703 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 30 01:42:40.054703 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 30 01:42:40.054703 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 30 01:42:40.054703 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 30 01:42:40.054703 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Aug 30 01:42:40.057153 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 124/251] 10.222.0.27 () {56 vars in 1096 bytes} [Tue Aug 30 01:42:39 2022] GET /identity/v3/roles/admin => generated 83 bytes in 222 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 01:42:40.058669 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72977cdc-a597-4977-a5d5-d19297c6b29c None None] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:40.059306 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72977cdc-a597-4977-a5d5-d19297c6b29c None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:40.059831 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72977cdc-a597-4977-a5d5-d19297c6b29c None None] PATH_INFO: `/` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:40.062269 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 125/252] 10.222.0.27 () {54 vars in 927 bytes} [Tue Aug 30 01:42:40 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 01:42:40.063370 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72977cdc-a597-4977-a5d5-d19297c6b29c None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:42:40.097505 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9de9235f-7709-46b3-a7a7-8ae6062a253f None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:42:40.137485 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b9c2fb5-4e23-4207-a8c2-fd8cf9a32da6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:42:40.138023 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b9c2fb5-4e23-4207-a8c2-fd8cf9a32da6 None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:40.138305 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b9c2fb5-4e23-4207-a8c2-fd8cf9a32da6 None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:40.138577 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b9c2fb5-4e23-4207-a8c2-fd8cf9a32da6 None admin] PATH_INFO: `/v3/roles` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:40.139250 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b9c2fb5-4e23-4207-a8c2-fd8cf9a32da6 None admin] RBAC: Adding query filter params (name=member) {{(pid=81578) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:42:40.139568 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b9c2fb5-4e23-4207-a8c2-fd8cf9a32da6 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:42:40.141096 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b9c2fb5-4e23-4207-a8c2-fd8cf9a32da6 None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:42:40.142652 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:42:40.151677 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 128/253] 10.222.0.27 () {56 vars in 1101 bytes} [Tue Aug 30 01:42:39 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:42:40.152606 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b9c2fb5-4e23-4207-a8c2-fd8cf9a32da6 None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:42:40.174820 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:42:40.183290 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72977cdc-a597-4977-a5d5-d19297c6b29c None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:42:40.224764 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36415bae-39dd-471d-9669-d958f814a0a1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:42:40.225591 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36415bae-39dd-471d-9669-d958f814a0a1 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:40.225973 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36415bae-39dd-471d-9669-d958f814a0a1 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:40.226713 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36415bae-39dd-471d-9669-d958f814a0a1 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:40.228155 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36415bae-39dd-471d-9669-d958f814a0a1 None admin] RBAC: Adding query filter params (role.id=ffde3a55b5dd43149c18c72b7f2fe6ae, scope.project.id=67913d5682a443069a5eed96f179751f, user.id=2ed4eca98551441e8a808afce7833f89) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:42:40.229023 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36415bae-39dd-471d-9669-d958f814a0a1 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:42:40.230471 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36415bae-39dd-471d-9669-d958f814a0a1 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:42:40.244763 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 126/254] 10.222.0.27 () {56 vars in 1374 bytes} [Tue Aug 30 01:42:40 2022] GET /identity/v3/role_assignments?user.id=2ed4eca98551441e8a808afce7833f89&scope.project.id=67913d5682a443069a5eed96f179751f&role.id=ffde3a55b5dd43149c18c72b7f2fe6ae => generated 259 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:42:40.245564 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7207796d-1c53-48b2-b2dc-67f4ce5338ff None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:40.245851 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7207796d-1c53-48b2-b2dc-67f4ce5338ff None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:40.246122 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7207796d-1c53-48b2-b2dc-67f4ce5338ff None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:40.267921 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-7207796d-1c53-48b2-b2dc-67f4ce5338ff None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 01:42:40.269880 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-7207796d-1c53-48b2-b2dc-67f4ce5338ff None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:42:40.325994 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b9c2fb5-4e23-4207-a8c2-fd8cf9a32da6 None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:42:40.354309 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 127/255] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Aug 30 01:42:40 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 30 01:42:40.355188 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7207796d-1c53-48b2-b2dc-67f4ce5338ff None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:42:40.423914 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1854bc3e-92c7-4ad3-9b7c-51599cf44994 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:42:40.424515 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1854bc3e-92c7-4ad3-9b7c-51599cf44994 None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:40.424801 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1854bc3e-92c7-4ad3-9b7c-51599cf44994 None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:40.425080 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1854bc3e-92c7-4ad3-9b7c-51599cf44994 None admin] PATH_INFO: `/v3/roles` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:40.425805 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1854bc3e-92c7-4ad3-9b7c-51599cf44994 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81578) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:42:40.426116 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1854bc3e-92c7-4ad3-9b7c-51599cf44994 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:42:40.427728 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1854bc3e-92c7-4ad3-9b7c-51599cf44994 None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:42:40.444994 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 129/256] 10.222.0.27 () {56 vars in 1099 bytes} [Tue Aug 30 01:42:40 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 293 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:42:40.451070 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1854bc3e-92c7-4ad3-9b7c-51599cf44994 None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:42:40.462158 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:42:40.475302 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7207796d-1c53-48b2-b2dc-67f4ce5338ff None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:42:40.525726 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a8e84a5-5ab6-41fa-b6ec-0e5627792211 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:42:40.526484 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a8e84a5-5ab6-41fa-b6ec-0e5627792211 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:40.526822 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a8e84a5-5ab6-41fa-b6ec-0e5627792211 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:40.527202 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a8e84a5-5ab6-41fa-b6ec-0e5627792211 None admin] PATH_INFO: `/v3/users/b3189d44e5c846e88be8df57633d93cd` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:40.528403 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a8e84a5-5ab6-41fa-b6ec-0e5627792211 None admin] RBAC: Authorizing `identity:get_user(user_id=b3189d44e5c846e88be8df57633d93cd)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:42:40.530270 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a8e84a5-5ab6-41fa-b6ec-0e5627792211 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:42:40.531937 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 128/257] 10.222.0.27 () {56 vars in 1177 bytes} [Tue Aug 30 01:42:40 2022] GET /identity/v3/users/b3189d44e5c846e88be8df57633d93cd => generated 297 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:42:40.537380 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a8e84a5-5ab6-41fa-b6ec-0e5627792211 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:42:40.555005 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:42:40.602867 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1854bc3e-92c7-4ad3-9b7c-51599cf44994 None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:42:40.639491 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3d8bc3d-da76-4c5f-9a8d-64e57a51d1cf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:42:40.640393 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3d8bc3d-da76-4c5f-9a8d-64e57a51d1cf None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:40.640822 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3d8bc3d-da76-4c5f-9a8d-64e57a51d1cf None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:40.641284 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3d8bc3d-da76-4c5f-9a8d-64e57a51d1cf None admin] PATH_INFO: `/v3/users/a4a8afdab07d4fd089132965087a3d56` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:40.642688 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3d8bc3d-da76-4c5f-9a8d-64e57a51d1cf None admin] RBAC: Authorizing `identity:get_user(user_id=a4a8afdab07d4fd089132965087a3d56)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:42:40.644345 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3d8bc3d-da76-4c5f-9a8d-64e57a51d1cf None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:42:40.646653 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 130/258] 10.222.0.27 () {56 vars in 1177 bytes} [Tue Aug 30 01:42:40 2022] GET /identity/v3/users/a4a8afdab07d4fd089132965087a3d56 => generated 283 bytes in 196 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 30 01:42:40.652601 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3d8bc3d-da76-4c5f-9a8d-64e57a51d1cf None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:42:40.706879 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a8e84a5-5ab6-41fa-b6ec-0e5627792211 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:42:40.772451 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfd8e999-9fa7-490e-9a9b-f73cbd09b96b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:42:40.773094 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfd8e999-9fa7-490e-9a9b-f73cbd09b96b None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:40.773374 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfd8e999-9fa7-490e-9a9b-f73cbd09b96b None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:40.773670 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfd8e999-9fa7-490e-9a9b-f73cbd09b96b None admin] PATH_INFO: `/v3/projects/67913d5682a443069a5eed96f179751f` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:40.776169 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfd8e999-9fa7-490e-9a9b-f73cbd09b96b None admin] RBAC: Authorizing `identity:get_project(project_id=67913d5682a443069a5eed96f179751f)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:42:40.777908 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfd8e999-9fa7-490e-9a9b-f73cbd09b96b None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:42:40.780263 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 129/259] 10.222.0.27 () {56 vars in 1186 bytes} [Tue Aug 30 01:42:40 2022] GET /identity/v3/projects/67913d5682a443069a5eed96f179751f => generated 302 bytes in 243 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 30 01:42:40.793264 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfd8e999-9fa7-490e-9a9b-f73cbd09b96b None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:42:40.840179 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3d8bc3d-da76-4c5f-9a8d-64e57a51d1cf None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:42:40.849486 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:42:40.913747 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:42:40.932332 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a28994d-b6cb-4ce3-9a1f-112a51860d25 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:42:40.933372 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a28994d-b6cb-4ce3-9a1f-112a51860d25 None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:40.938019 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a28994d-b6cb-4ce3-9a1f-112a51860d25 None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:40.938506 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a28994d-b6cb-4ce3-9a1f-112a51860d25 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:41.004173 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfd8e999-9fa7-490e-9a9b-f73cbd09b96b None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:42:41.047113 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72c247dd-5f47-4069-8d17-5f9a55f746fa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:42:41.047780 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72c247dd-5f47-4069-8d17-5f9a55f746fa None admin] REQUEST_METHOD: `PUT` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:41.048060 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72c247dd-5f47-4069-8d17-5f9a55f746fa None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:41.048332 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72c247dd-5f47-4069-8d17-5f9a55f746fa None admin] PATH_INFO: `/v3/projects/67913d5682a443069a5eed96f179751f/users/b3189d44e5c846e88be8df57633d93cd/roles/1fb81afcb9f245f3b989add94598a063` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:41.050831 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72c247dd-5f47-4069-8d17-5f9a55f746fa None admin] RBAC: Authorizing `identity:create_grant(project_id=67913d5682a443069a5eed96f179751f, user_id=b3189d44e5c846e88be8df57633d93cd, role_id=1fb81afcb9f245f3b989add94598a063)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:42:41.052734 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72c247dd-5f47-4069-8d17-5f9a55f746fa None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:42:41.073823 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a28994d-b6cb-4ce3-9a1f-112a51860d25 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:42:41.075866 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a28994d-b6cb-4ce3-9a1f-112a51860d25 None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:42:41.144735 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 130/260] 10.222.0.27 () {58 vars in 1439 bytes} [Tue Aug 30 01:42:40 2022] PUT /identity/v3/projects/67913d5682a443069a5eed96f179751f/users/b3189d44e5c846e88be8df57633d93cd/roles/1fb81afcb9f245f3b989add94598a063 => generated 0 bytes in 353 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 30 01:42:41.145403 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72c247dd-5f47-4069-8d17-5f9a55f746fa None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:42:41.199890 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 131/261] 10.222.0.27 () {58 vars in 1282 bytes} [Tue Aug 30 01:42:40 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 547 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Aug 30 01:42:41.207018 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a28994d-b6cb-4ce3-9a1f-112a51860d25 None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:42:41.280569 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:42:41.359972 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:42:41.382054 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72c247dd-5f47-4069-8d17-5f9a55f746fa None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:42:41.425519 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a28994d-b6cb-4ce3-9a1f-112a51860d25 None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:42:41.439620 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3348f7d4-8653-4de2-b0e0-dce86e708046 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:42:41.440290 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3348f7d4-8653-4de2-b0e0-dce86e708046 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:41.440575 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3348f7d4-8653-4de2-b0e0-dce86e708046 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:41.440847 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3348f7d4-8653-4de2-b0e0-dce86e708046 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:41.461129 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3348f7d4-8653-4de2-b0e0-dce86e708046 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 30 01:42:41.466562 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application [None req-3348f7d4-8653-4de2-b0e0-dce86e708046 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Aug 30 01:42:41.466562 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 01:42:41.466562 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 01:42:41.466562 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 01:42:41.466562 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 01:42:41.466562 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 01:42:41.466562 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 01:42:41.466562 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 01:42:41.466562 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 01:42:41.466562 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 01:42:41.466562 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 01:42:41.466562 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 01:42:41.466562 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 30 01:42:41.466562 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 30 01:42:41.466562 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 30 01:42:41.466562 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 30 01:42:41.467452 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 30 01:42:41.467452 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 30 01:42:41.467452 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 30 01:42:41.467452 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 30 01:42:41.467452 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 01:42:41.467452 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 01:42:41.467452 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 01:42:41.467452 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 01:42:41.467452 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 01:42:41.467452 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 01:42:41.467452 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 01:42:41.467452 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application with Lock( Aug 30 01:42:41.467452 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 01:42:41.467452 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._enter() Aug 30 01:42:41.467452 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 01:42:41.467452 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 01:42:41.468285 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 01:42:41.468285 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.creator() Aug 30 01:42:41.468285 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 01:42:41.468285 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application created_value = creator( Aug 30 01:42:41.468285 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 30 01:42:41.468285 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 30 01:42:41.468285 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 30 01:42:41.468285 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 30 01:42:41.468285 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 30 01:42:41.468285 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 30 01:42:41.468285 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Aug 30 01:42:41.468285 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Aug 30 01:42:41.468285 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 131/262] 10.222.0.27 () {56 vars in 1114 bytes} [Tue Aug 30 01:42:41 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 323 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 01:42:41.473934 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3348f7d4-8653-4de2-b0e0-dce86e708046 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:42:41.498480 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-859d90ca-5e57-477c-9ef3-dd064c5f8732 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:42:41.500173 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-859d90ca-5e57-477c-9ef3-dd064c5f8732 None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:41.500651 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-859d90ca-5e57-477c-9ef3-dd064c5f8732 None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:41.501111 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-859d90ca-5e57-477c-9ef3-dd064c5f8732 None admin] PATH_INFO: `/v3/projects/67913d5682a443069a5eed96f179751f` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:41.502584 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-859d90ca-5e57-477c-9ef3-dd064c5f8732 None admin] RBAC: Authorizing `identity:get_project(project_id=67913d5682a443069a5eed96f179751f)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:42:41.505153 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-859d90ca-5e57-477c-9ef3-dd064c5f8732 None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:42:41.508009 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 132/263] 10.222.0.27 () {56 vars in 1186 bytes} [Tue Aug 30 01:42:41 2022] GET /identity/v3/projects/67913d5682a443069a5eed96f179751f => generated 302 bytes in 302 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:42:41.517074 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-859d90ca-5e57-477c-9ef3-dd064c5f8732 None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:42:41.647796 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3348f7d4-8653-4de2-b0e0-dce86e708046 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:42:41.655980 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:42:41.663339 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-859d90ca-5e57-477c-9ef3-dd064c5f8732 None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:42:41.687611 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fbbb0a0-2cc1-42d8-9347-b17a681dd674 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:42:41.688574 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fbbb0a0-2cc1-42d8-9347-b17a681dd674 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:41.689047 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fbbb0a0-2cc1-42d8-9347-b17a681dd674 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:41.689604 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fbbb0a0-2cc1-42d8-9347-b17a681dd674 None admin] PATH_INFO: `/v3/roles` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:41.690464 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fbbb0a0-2cc1-42d8-9347-b17a681dd674 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:42:41.691086 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fbbb0a0-2cc1-42d8-9347-b17a681dd674 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:42:41.693062 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fbbb0a0-2cc1-42d8-9347-b17a681dd674 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:42:41.700443 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37136c8a-7a6c-48c6-8b66-47194d38ed5d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:42:41.701101 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37136c8a-7a6c-48c6-8b66-47194d38ed5d None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:41.701415 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37136c8a-7a6c-48c6-8b66-47194d38ed5d None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:41.703464 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37136c8a-7a6c-48c6-8b66-47194d38ed5d None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:41.703464 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37136c8a-7a6c-48c6-8b66-47194d38ed5d None admin] RBAC: Adding query filter params (role.id=ead5fac86a0747ea8823a82efcea8ed7, scope.project.id=67913d5682a443069a5eed96f179751f, user.id=a4a8afdab07d4fd089132965087a3d56) {{(pid=81578) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:42:41.703464 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37136c8a-7a6c-48c6-8b66-47194d38ed5d None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:42:41.705305 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37136c8a-7a6c-48c6-8b66-47194d38ed5d None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:42:41.709890 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 132/264] 10.222.0.27 () {56 vars in 1111 bytes} [Tue Aug 30 01:42:41 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 236 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:42:41.710713 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 133/265] 10.222.0.27 () {56 vars in 1374 bytes} [Tue Aug 30 01:42:41 2022] GET /identity/v3/role_assignments?user.id=a4a8afdab07d4fd089132965087a3d56&scope.project.id=67913d5682a443069a5eed96f179751f&role.id=ead5fac86a0747ea8823a82efcea8ed7 => generated 608 bytes in 194 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:42:41.729284 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fbbb0a0-2cc1-42d8-9347-b17a681dd674 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:42:41.739675 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:42:41.857840 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fbbb0a0-2cc1-42d8-9347-b17a681dd674 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:42:41.893874 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e97fa7bd-033a-4257-b3a9-359d6328e36b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:42:41.894968 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e97fa7bd-033a-4257-b3a9-359d6328e36b None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:41.895453 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e97fa7bd-033a-4257-b3a9-359d6328e36b None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:41.895919 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e97fa7bd-033a-4257-b3a9-359d6328e36b None admin] PATH_INFO: `/v3/users/a4a8afdab07d4fd089132965087a3d56` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:41.897415 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e97fa7bd-033a-4257-b3a9-359d6328e36b None admin] RBAC: Authorizing `identity:get_user(user_id=a4a8afdab07d4fd089132965087a3d56)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:42:41.899353 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e97fa7bd-033a-4257-b3a9-359d6328e36b None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:42:41.901699 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 133/266] 10.222.0.27 () {56 vars in 1177 bytes} [Tue Aug 30 01:42:41 2022] GET /identity/v3/users/a4a8afdab07d4fd089132965087a3d56 => generated 283 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:42:41.907099 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37136c8a-7a6c-48c6-8b66-47194d38ed5d None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:42:42.013209 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb9e8eb7-5190-4577-b237-e92a4431d4e0 None None] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:42.014351 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb9e8eb7-5190-4577-b237-e92a4431d4e0 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:42.014843 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb9e8eb7-5190-4577-b237-e92a4431d4e0 None None] PATH_INFO: `/` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:42.016721 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 134/267] 10.222.0.27 () {54 vars in 927 bytes} [Tue Aug 30 01:42:42 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 01:42:42.023468 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8eef510d-50c2-4060-a234-e9ca975a3648 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:42.024665 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8eef510d-50c2-4060-a234-e9ca975a3648 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:42.025427 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8eef510d-50c2-4060-a234-e9ca975a3648 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:42.052913 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37136c8a-7a6c-48c6-8b66-47194d38ed5d None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:42:42.059467 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-8eef510d-50c2-4060-a234-e9ca975a3648 None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 01:42:42.061018 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-8eef510d-50c2-4060-a234-e9ca975a3648 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:42:42.093491 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f03770a-7d82-4fae-96a0-66a563567e3b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:42:42.094104 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f03770a-7d82-4fae-96a0-66a563567e3b None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:42.094383 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f03770a-7d82-4fae-96a0-66a563567e3b None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:42.094657 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f03770a-7d82-4fae-96a0-66a563567e3b None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:42.127376 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:42:42.147311 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 135/268] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Aug 30 01:42:42 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 125 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Aug 30 01:42:42.177339 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f03770a-7d82-4fae-96a0-66a563567e3b None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:42:42.178767 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f03770a-7d82-4fae-96a0-66a563567e3b None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:42:42.299376 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 134/269] 10.222.0.27 () {58 vars in 1282 bytes} [Tue Aug 30 01:42:41 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 393 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 30 01:42:42.305017 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8eef510d-50c2-4060-a234-e9ca975a3648 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:42:42.313546 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:42:42.446982 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8eef510d-50c2-4060-a234-e9ca975a3648 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:42:42.511110 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32ed3b46-95d8-4bab-b993-176de7ab047e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:42:42.511768 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32ed3b46-95d8-4bab-b993-176de7ab047e None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:42.512046 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32ed3b46-95d8-4bab-b993-176de7ab047e None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:42.512314 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32ed3b46-95d8-4bab-b993-176de7ab047e None admin] PATH_INFO: `/v3/projects/67913d5682a443069a5eed96f179751f` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:42.513450 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32ed3b46-95d8-4bab-b993-176de7ab047e None admin] RBAC: Authorizing `identity:get_project(project_id=67913d5682a443069a5eed96f179751f)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:42:42.514995 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32ed3b46-95d8-4bab-b993-176de7ab047e None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:42:42.517069 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 136/270] 10.222.0.27 () {56 vars in 1186 bytes} [Tue Aug 30 01:42:42 2022] GET /identity/v3/projects/67913d5682a443069a5eed96f179751f => generated 302 bytes in 212 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:42:42.522828 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f03770a-7d82-4fae-96a0-66a563567e3b None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:42:42.594748 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:42:42.601648 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32ed3b46-95d8-4bab-b993-176de7ab047e None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:42:42.651701 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f03770a-7d82-4fae-96a0-66a563567e3b None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:42:42.687020 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:42:42.701258 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3acbf319-b73e-4434-807c-d66c713e2d9f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:42:42.701258 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3acbf319-b73e-4434-807c-d66c713e2d9f None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:42.701258 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3acbf319-b73e-4434-807c-d66c713e2d9f None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:42.701258 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3acbf319-b73e-4434-807c-d66c713e2d9f None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:42.701692 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3acbf319-b73e-4434-807c-d66c713e2d9f None admin] RBAC: Adding query filter params (role.id=4621068972ba4f79bad104f956db04aa, scope.project.id=67913d5682a443069a5eed96f179751f, user.id=a4a8afdab07d4fd089132965087a3d56) {{(pid=81578) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:42:42.701692 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3acbf319-b73e-4434-807c-d66c713e2d9f None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:42:42.707856 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3acbf319-b73e-4434-807c-d66c713e2d9f None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:42:42.715447 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 135/271] 10.222.0.27 () {56 vars in 1374 bytes} [Tue Aug 30 01:42:42 2022] GET /identity/v3/role_assignments?user.id=a4a8afdab07d4fd089132965087a3d56&scope.project.id=67913d5682a443069a5eed96f179751f&role.id=4621068972ba4f79bad104f956db04aa => generated 608 bytes in 193 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:42:42.723630 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3842478f-f575-4a06-af28-e16ee0e6d9c4 None None] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:42.724148 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3842478f-f575-4a06-af28-e16ee0e6d9c4 None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:42.724565 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3842478f-f575-4a06-af28-e16ee0e6d9c4 None None] PATH_INFO: `/` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:42.726431 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 136/272] 10.222.0.27 () {54 vars in 927 bytes} [Tue Aug 30 01:42:42 2022] GET /identity => generated 270 bytes in 10 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 01:42:42.733535 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32ed3b46-95d8-4bab-b993-176de7ab047e None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:42:42.733726 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-500bd157-c09b-444d-8819-ccedd4254164 None None] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:42.733726 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-500bd157-c09b-444d-8819-ccedd4254164 None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:42.733726 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-500bd157-c09b-444d-8819-ccedd4254164 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:42.789106 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.auth.core [None req-500bd157-c09b-444d-8819-ccedd4254164 None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81578) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 01:42:42.792596 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5eb156e-1773-4a87-9cce-02e764fafb4e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:42:42.793521 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5eb156e-1773-4a87-9cce-02e764fafb4e None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:42.793978 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5eb156e-1773-4a87-9cce-02e764fafb4e None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:42.794394 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5eb156e-1773-4a87-9cce-02e764fafb4e None admin] PATH_INFO: `/v3/roles/member` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:42.795193 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-500bd157-c09b-444d-8819-ccedd4254164 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:42:42.824903 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5eb156e-1773-4a87-9cce-02e764fafb4e None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 30 01:42:42.835110 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application [None req-a5eb156e-1773-4a87-9cce-02e764fafb4e None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Aug 30 01:42:42.835110 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 01:42:42.835110 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 01:42:42.835110 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 01:42:42.835110 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 01:42:42.835110 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 01:42:42.835110 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 01:42:42.835110 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 01:42:42.835110 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 01:42:42.835110 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 01:42:42.835110 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 01:42:42.835110 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 01:42:42.835110 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 30 01:42:42.835110 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 30 01:42:42.835110 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 30 01:42:42.835110 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 30 01:42:42.836287 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 30 01:42:42.836287 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 30 01:42:42.836287 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 30 01:42:42.836287 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 30 01:42:42.836287 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 01:42:42.836287 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 01:42:42.836287 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 01:42:42.836287 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 01:42:42.836287 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 01:42:42.836287 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 01:42:42.836287 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 01:42:42.836287 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application with Lock( Aug 30 01:42:42.836287 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 01:42:42.836287 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._enter() Aug 30 01:42:42.836287 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 01:42:42.836287 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 01:42:42.837931 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 01:42:42.837931 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.creator() Aug 30 01:42:42.837931 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 01:42:42.837931 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application created_value = creator( Aug 30 01:42:42.837931 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 30 01:42:42.837931 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 30 01:42:42.837931 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 30 01:42:42.837931 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 30 01:42:42.837931 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 30 01:42:42.837931 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 30 01:42:42.837931 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Aug 30 01:42:42.837931 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Aug 30 01:42:42.843865 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 137/273] 10.222.0.27 () {56 vars in 1099 bytes} [Tue Aug 30 01:42:42 2022] GET /identity/v3/roles/member => generated 84 bytes in 243 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 30 01:42:42.850438 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5eb156e-1773-4a87-9cce-02e764fafb4e None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:42:42.953400 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 137/274] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Aug 30 01:42:42 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 222 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Aug 30 01:42:42.995662 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5eb156e-1773-4a87-9cce-02e764fafb4e None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:42:43.028150 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:42:43.037309 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9e42230-3a11-4ca4-9b7d-d52077bc15c6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:42:43.038010 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9e42230-3a11-4ca4-9b7d-d52077bc15c6 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:43.038298 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9e42230-3a11-4ca4-9b7d-d52077bc15c6 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:43.038569 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9e42230-3a11-4ca4-9b7d-d52077bc15c6 None admin] PATH_INFO: `/v3/roles` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:43.039246 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9e42230-3a11-4ca4-9b7d-d52077bc15c6 None admin] RBAC: Adding query filter params (name=member) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:42:43.039584 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9e42230-3a11-4ca4-9b7d-d52077bc15c6 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:42:43.041190 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9e42230-3a11-4ca4-9b7d-d52077bc15c6 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:42:43.051625 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 138/275] 10.222.0.27 () {56 vars in 1101 bytes} [Tue Aug 30 01:42:42 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 202 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:42:43.057957 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-500bd157-c09b-444d-8819-ccedd4254164 None None] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:42:43.099764 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19f9a267-ca2c-4f44-a71f-23d1ae39ffe8 None None] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:43.100051 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19f9a267-ca2c-4f44-a71f-23d1ae39ffe8 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:43.100316 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19f9a267-ca2c-4f44-a71f-23d1ae39ffe8 None None] PATH_INFO: `/` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:43.102432 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 139/276] 10.222.0.27 () {54 vars in 927 bytes} [Tue Aug 30 01:42:43 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 30 01:42:43.108237 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a511ed2f-f350-4acd-b8df-ca4e6e39f62d None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:43.108531 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a511ed2f-f350-4acd-b8df-ca4e6e39f62d None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:43.108794 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a511ed2f-f350-4acd-b8df-ca4e6e39f62d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:43.126402 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:42:43.138225 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-a511ed2f-f350-4acd-b8df-ca4e6e39f62d None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 01:42:43.139963 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-a511ed2f-f350-4acd-b8df-ca4e6e39f62d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:42:43.196977 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-500bd157-c09b-444d-8819-ccedd4254164 None None] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:42:43.232407 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 140/277] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Aug 30 01:42:43 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 125 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Aug 30 01:42:43.239763 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37f1f146-09bb-4ad0-a942-422736f0f313 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:42:43.240445 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37f1f146-09bb-4ad0-a942-422736f0f313 None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:43.240725 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37f1f146-09bb-4ad0-a942-422736f0f313 None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:43.240996 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37f1f146-09bb-4ad0-a942-422736f0f313 None admin] PATH_INFO: `/v3/users/87b7b50b8f3547969f5ecaaed9ac936d` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:43.257413 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37f1f146-09bb-4ad0-a942-422736f0f313 None admin] RBAC: Authorizing `identity:get_user(user_id=87b7b50b8f3547969f5ecaaed9ac936d)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:42:43.259013 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37f1f146-09bb-4ad0-a942-422736f0f313 None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:42:43.261023 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 138/278] 10.222.0.27 () {56 vars in 1177 bytes} [Tue Aug 30 01:42:43 2022] GET /identity/v3/users/87b7b50b8f3547969f5ecaaed9ac936d => generated 293 bytes in 204 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:42:43.266715 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a511ed2f-f350-4acd-b8df-ca4e6e39f62d None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:42:43.278179 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37f1f146-09bb-4ad0-a942-422736f0f313 None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:42:43.396699 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a511ed2f-f350-4acd-b8df-ca4e6e39f62d None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:42:43.404004 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37f1f146-09bb-4ad0-a942-422736f0f313 None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:42:43.419899 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:42:43.425005 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:42:43.443476 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67ebec0a-ddfc-49b3-b3b0-86fc6d1b1fdc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:42:43.444101 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67ebec0a-ddfc-49b3-b3b0-86fc6d1b1fdc None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:43.444380 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67ebec0a-ddfc-49b3-b3b0-86fc6d1b1fdc None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:43.444649 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67ebec0a-ddfc-49b3-b3b0-86fc6d1b1fdc None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:43.445741 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67ebec0a-ddfc-49b3-b3b0-86fc6d1b1fdc None admin] RBAC: Adding query filter params (role.id=1fb81afcb9f245f3b989add94598a063, user.id=87b7b50b8f3547969f5ecaaed9ac936d, scope.system=all) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:42:43.446059 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67ebec0a-ddfc-49b3-b3b0-86fc6d1b1fdc None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:42:43.449626 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67ebec0a-ddfc-49b3-b3b0-86fc6d1b1fdc None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:42:43.473657 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-504f30eb-243a-4408-9a39-ada59e05e5a7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:42:43.473657 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-504f30eb-243a-4408-9a39-ada59e05e5a7 None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:43.473657 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-504f30eb-243a-4408-9a39-ada59e05e5a7 None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:43.473657 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-504f30eb-243a-4408-9a39-ada59e05e5a7 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:43.474192 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-504f30eb-243a-4408-9a39-ada59e05e5a7 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:42:43.474192 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-504f30eb-243a-4408-9a39-ada59e05e5a7 None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:42:43.474338 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 141/280] 10.222.0.27 () {56 vars in 1308 bytes} [Tue Aug 30 01:42:43 2022] GET /identity/v3/role_assignments?user.id=87b7b50b8f3547969f5ecaaed9ac936d&scope.system=all&role.id=1fb81afcb9f245f3b989add94598a063 => generated 226 bytes in 190 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:42:43.474409 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 139/280] 10.222.0.27 () {56 vars in 1108 bytes} [Tue Aug 30 01:42:43 2022] GET /identity/v3/domains/default => generated 199 bytes in 183 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 30 01:42:43.474474 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67ebec0a-ddfc-49b3-b3b0-86fc6d1b1fdc None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:42:43.520372 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-504f30eb-243a-4408-9a39-ada59e05e5a7 None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:42:43.607463 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67ebec0a-ddfc-49b3-b3b0-86fc6d1b1fdc None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:42:43.654826 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95618af0-3501-4259-975e-8ec0aa779858 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:42:43.655293 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-504f30eb-243a-4408-9a39-ada59e05e5a7 None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:42:43.656373 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95618af0-3501-4259-975e-8ec0aa779858 None admin] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:43.656957 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95618af0-3501-4259-975e-8ec0aa779858 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:43.657447 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95618af0-3501-4259-975e-8ec0aa779858 None admin] PATH_INFO: `/v3/users` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:43.658470 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95618af0-3501-4259-975e-8ec0aa779858 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:42:43.660335 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95618af0-3501-4259-975e-8ec0aa779858 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:42:43.693970 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-525c362f-2400-42e1-93a2-0347c3f7e443 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:42:43.694658 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-525c362f-2400-42e1-93a2-0347c3f7e443 None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:43.694938 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-525c362f-2400-42e1-93a2-0347c3f7e443 None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:43.695206 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-525c362f-2400-42e1-93a2-0347c3f7e443 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:43.706459 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-525c362f-2400-42e1-93a2-0347c3f7e443 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 30 01:42:43.711994 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application [None req-525c362f-2400-42e1-93a2-0347c3f7e443 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Aug 30 01:42:43.711994 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 01:42:43.711994 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 01:42:43.711994 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 01:42:43.711994 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 01:42:43.711994 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 01:42:43.711994 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 01:42:43.711994 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 01:42:43.711994 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 01:42:43.711994 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 01:42:43.711994 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 01:42:43.711994 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 01:42:43.711994 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 30 01:42:43.711994 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 30 01:42:43.711994 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 30 01:42:43.711994 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 30 01:42:43.712890 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 30 01:42:43.712890 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 30 01:42:43.712890 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 30 01:42:43.712890 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 30 01:42:43.712890 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 01:42:43.712890 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 01:42:43.712890 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 01:42:43.712890 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 01:42:43.712890 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 01:42:43.712890 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 01:42:43.712890 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 01:42:43.712890 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application with Lock( Aug 30 01:42:43.712890 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 01:42:43.712890 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._enter() Aug 30 01:42:43.712890 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 01:42:43.712890 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 01:42:43.714368 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 01:42:43.714368 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.creator() Aug 30 01:42:43.714368 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 01:42:43.714368 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application created_value = creator( Aug 30 01:42:43.714368 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 30 01:42:43.714368 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 30 01:42:43.714368 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 30 01:42:43.714368 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 30 01:42:43.714368 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 30 01:42:43.714368 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 30 01:42:43.714368 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Aug 30 01:42:43.714368 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application Aug 30 01:42:43.714368 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 140/281] 10.222.0.27 () {56 vars in 1099 bytes} [Tue Aug 30 01:42:43 2022] GET /identity/v3/roles/reader => generated 84 bytes in 194 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 30 01:42:43.725351 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-525c362f-2400-42e1-93a2-0347c3f7e443 None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:42:43.765904 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:42:43.798388 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 142/282] 10.222.0.27 () {60 vars in 1132 bytes} [Tue Aug 30 01:42:43 2022] POST /identity/v3/users => generated 293 bytes in 331 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 01:42:43.839358 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-525c362f-2400-42e1-93a2-0347c3f7e443 None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:42:43.881754 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7dd3a6fb-90ee-429c-869e-ba546ce97bdd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:42:43.882376 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dd3a6fb-90ee-429c-869e-ba546ce97bdd None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:43.882658 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dd3a6fb-90ee-429c-869e-ba546ce97bdd None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:43.882930 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dd3a6fb-90ee-429c-869e-ba546ce97bdd None admin] PATH_INFO: `/v3/roles` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:43.884479 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7dd3a6fb-90ee-429c-869e-ba546ce97bdd None admin] RBAC: Adding query filter params (name=reader) {{(pid=81578) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:42:43.884814 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7dd3a6fb-90ee-429c-869e-ba546ce97bdd None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:42:43.886264 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7dd3a6fb-90ee-429c-869e-ba546ce97bdd None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:42:43.896943 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 141/283] 10.222.0.27 () {56 vars in 1101 bytes} [Tue Aug 30 01:42:43 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:42:43.902967 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95618af0-3501-4259-975e-8ec0aa779858 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:42:43.965829 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e503ed44-7c68-4014-9e54-6134f0b07ce6 None None] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:43.966171 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e503ed44-7c68-4014-9e54-6134f0b07ce6 None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:43.966377 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e503ed44-7c68-4014-9e54-6134f0b07ce6 None None] PATH_INFO: `/` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:43.968336 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 142/284] 10.222.0.27 () {54 vars in 927 bytes} [Tue Aug 30 01:42:43 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 30 01:42:43.973719 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4aa3bca-0545-4527-bdf1-63d57ffb143d None None] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:43.974007 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4aa3bca-0545-4527-bdf1-63d57ffb143d None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:43.974314 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4aa3bca-0545-4527-bdf1-63d57ffb143d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:43.977949 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:42:43.999973 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.auth.core [None req-d4aa3bca-0545-4527-bdf1-63d57ffb143d None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81578) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 01:42:44.001224 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-d4aa3bca-0545-4527-bdf1-63d57ffb143d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:42:44.046912 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95618af0-3501-4259-975e-8ec0aa779858 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:42:44.090778 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d80ca611-33de-4e03-9c64-de02e7d90cb8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:42:44.092321 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 143/285] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Aug 30 01:42:43 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 117 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Aug 30 01:42:44.094820 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d80ca611-33de-4e03-9c64-de02e7d90cb8 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:44.094820 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d80ca611-33de-4e03-9c64-de02e7d90cb8 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:44.094820 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d80ca611-33de-4e03-9c64-de02e7d90cb8 None admin] PATH_INFO: `/v3/users/2ed4eca98551441e8a808afce7833f89` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:44.098506 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d80ca611-33de-4e03-9c64-de02e7d90cb8 None admin] RBAC: Authorizing `identity:get_user(user_id=2ed4eca98551441e8a808afce7833f89)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:42:44.100183 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d80ca611-33de-4e03-9c64-de02e7d90cb8 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:42:44.104982 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 143/286] 10.222.0.27 () {56 vars in 1177 bytes} [Tue Aug 30 01:42:43 2022] GET /identity/v3/users/2ed4eca98551441e8a808afce7833f89 => generated 297 bytes in 201 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:42:44.113145 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4aa3bca-0545-4527-bdf1-63d57ffb143d None None] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:42:44.171209 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:42:44.250933 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4aa3bca-0545-4527-bdf1-63d57ffb143d None None] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:42:44.290914 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce4b7c3d-68ab-4633-9f96-7412c8094f19 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:42:44.291938 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce4b7c3d-68ab-4633-9f96-7412c8094f19 None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:44.292410 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce4b7c3d-68ab-4633-9f96-7412c8094f19 None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:44.292745 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce4b7c3d-68ab-4633-9f96-7412c8094f19 None admin] PATH_INFO: `/v3/projects/67913d5682a443069a5eed96f179751f` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:44.293904 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce4b7c3d-68ab-4633-9f96-7412c8094f19 None admin] RBAC: Authorizing `identity:get_project(project_id=67913d5682a443069a5eed96f179751f)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:42:44.295338 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce4b7c3d-68ab-4633-9f96-7412c8094f19 None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:42:44.297150 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 144/287] 10.222.0.27 () {56 vars in 1186 bytes} [Tue Aug 30 01:42:44 2022] GET /identity/v3/projects/67913d5682a443069a5eed96f179751f => generated 302 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:42:44.302886 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d80ca611-33de-4e03-9c64-de02e7d90cb8 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:42:44.443455 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d80ca611-33de-4e03-9c64-de02e7d90cb8 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:42:44.459044 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:42:44.489647 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a510b194-ca29-4d31-a768-a66d0058e6c6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:42:44.490408 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a510b194-ca29-4d31-a768-a66d0058e6c6 None admin] REQUEST_METHOD: `PUT` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:44.490754 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a510b194-ca29-4d31-a768-a66d0058e6c6 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:44.491094 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a510b194-ca29-4d31-a768-a66d0058e6c6 None admin] PATH_INFO: `/v3/projects/67913d5682a443069a5eed96f179751f/users/2ed4eca98551441e8a808afce7833f89/roles/ffde3a55b5dd43149c18c72b7f2fe6ae` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:44.496245 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a510b194-ca29-4d31-a768-a66d0058e6c6 None admin] RBAC: Authorizing `identity:create_grant(project_id=67913d5682a443069a5eed96f179751f, user_id=2ed4eca98551441e8a808afce7833f89, role_id=ffde3a55b5dd43149c18c72b7f2fe6ae)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:42:44.497820 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a510b194-ca29-4d31-a768-a66d0058e6c6 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:42:44.525508 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce4b7c3d-68ab-4633-9f96-7412c8094f19 None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:42:44.620788 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 144/288] 10.222.0.27 () {58 vars in 1439 bytes} [Tue Aug 30 01:42:44 2022] PUT /identity/v3/projects/67913d5682a443069a5eed96f179751f/users/2ed4eca98551441e8a808afce7833f89/roles/ffde3a55b5dd43149c18c72b7f2fe6ae => generated 0 bytes in 318 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 01:42:44.653857 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:42:44.658254 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce4b7c3d-68ab-4633-9f96-7412c8094f19 None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:42:44.725330 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab916565-27f8-4f53-a08f-a34e7b943ce4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:42:44.726450 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab916565-27f8-4f53-a08f-a34e7b943ce4 None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:44.726958 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab916565-27f8-4f53-a08f-a34e7b943ce4 None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:44.728067 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab916565-27f8-4f53-a08f-a34e7b943ce4 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:44.748999 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab916565-27f8-4f53-a08f-a34e7b943ce4 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 30 01:42:44.754249 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application [None req-ab916565-27f8-4f53-a08f-a34e7b943ce4 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Aug 30 01:42:44.754249 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 01:42:44.754249 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 01:42:44.754249 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 01:42:44.754249 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 01:42:44.754249 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 01:42:44.754249 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 01:42:44.754249 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 01:42:44.754249 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 01:42:44.754249 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 01:42:44.754249 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 01:42:44.754249 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 01:42:44.754249 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 30 01:42:44.754249 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 30 01:42:44.754249 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 30 01:42:44.754249 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 30 01:42:44.755305 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 30 01:42:44.755305 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 30 01:42:44.755305 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 30 01:42:44.755305 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 30 01:42:44.755305 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 01:42:44.755305 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 01:42:44.755305 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 01:42:44.755305 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 01:42:44.755305 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 01:42:44.755305 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 01:42:44.755305 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 01:42:44.755305 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application with Lock( Aug 30 01:42:44.755305 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 01:42:44.755305 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._enter() Aug 30 01:42:44.755305 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 01:42:44.755305 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 01:42:44.756298 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 01:42:44.756298 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.creator() Aug 30 01:42:44.756298 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 01:42:44.756298 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application created_value = creator( Aug 30 01:42:44.756298 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 30 01:42:44.756298 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 30 01:42:44.756298 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 30 01:42:44.756298 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 30 01:42:44.756298 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 30 01:42:44.756298 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 30 01:42:44.756298 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Aug 30 01:42:44.756298 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application Aug 30 01:42:44.758853 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 145/289] 10.222.0.27 () {56 vars in 1099 bytes} [Tue Aug 30 01:42:44 2022] GET /identity/v3/roles/member => generated 84 bytes in 235 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 01:42:44.764876 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a510b194-ca29-4d31-a768-a66d0058e6c6 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:42:44.901313 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a510b194-ca29-4d31-a768-a66d0058e6c6 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:42:44.939697 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c57d9b5d-e98b-43f6-8f59-440827ee06bb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:42:44.940507 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c57d9b5d-e98b-43f6-8f59-440827ee06bb None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:44.940885 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c57d9b5d-e98b-43f6-8f59-440827ee06bb None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:44.941248 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c57d9b5d-e98b-43f6-8f59-440827ee06bb None admin] PATH_INFO: `/v3/roles` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:44.942132 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c57d9b5d-e98b-43f6-8f59-440827ee06bb None admin] RBAC: Adding query filter params (name=member) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:42:44.942574 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c57d9b5d-e98b-43f6-8f59-440827ee06bb None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:42:44.944097 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c57d9b5d-e98b-43f6-8f59-440827ee06bb None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:42:44.953886 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 145/290] 10.222.0.27 () {56 vars in 1101 bytes} [Tue Aug 30 01:42:44 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 190 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:42:44.959720 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab916565-27f8-4f53-a08f-a34e7b943ce4 None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:42:45.079504 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab916565-27f8-4f53-a08f-a34e7b943ce4 None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:42:45.127280 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b943a725-97e3-4488-a3d2-86cc67e28352 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:42:45.127919 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b943a725-97e3-4488-a3d2-86cc67e28352 None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:45.128203 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b943a725-97e3-4488-a3d2-86cc67e28352 None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:45.128472 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b943a725-97e3-4488-a3d2-86cc67e28352 None admin] PATH_INFO: `/v3/users/b3189d44e5c846e88be8df57633d93cd` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:45.129656 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b943a725-97e3-4488-a3d2-86cc67e28352 None admin] RBAC: Authorizing `identity:get_user(user_id=b3189d44e5c846e88be8df57633d93cd)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:42:45.131105 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b943a725-97e3-4488-a3d2-86cc67e28352 None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:42:45.132912 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 146/291] 10.222.0.27 () {56 vars in 1177 bytes} [Tue Aug 30 01:42:44 2022] GET /identity/v3/users/b3189d44e5c846e88be8df57633d93cd => generated 297 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:42:45.139291 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c57d9b5d-e98b-43f6-8f59-440827ee06bb None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:42:45.187114 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:42:45.276870 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c57d9b5d-e98b-43f6-8f59-440827ee06bb None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:42:45.324200 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddb9f87d-dc20-473e-a5a4-6c4e61d41e31 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:42:45.324819 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddb9f87d-dc20-473e-a5a4-6c4e61d41e31 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:45.325100 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddb9f87d-dc20-473e-a5a4-6c4e61d41e31 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:45.325374 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddb9f87d-dc20-473e-a5a4-6c4e61d41e31 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:45.410864 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ddb9f87d-dc20-473e-a5a4-6c4e61d41e31 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:42:45.412240 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ddb9f87d-dc20-473e-a5a4-6c4e61d41e31 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:42:45.464991 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:42:45.535371 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 146/292] 10.222.0.27 () {58 vars in 1282 bytes} [Tue Aug 30 01:42:45 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 396 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 30 01:42:45.541593 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b943a725-97e3-4488-a3d2-86cc67e28352 None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:42:45.591854 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:42:45.671337 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f9b5208-64d2-4cea-a06a-9ccaa845f351 None None] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:45.671945 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f9b5208-64d2-4cea-a06a-9ccaa845f351 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:45.672421 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f9b5208-64d2-4cea-a06a-9ccaa845f351 None None] PATH_INFO: `/` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:45.677774 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 147/293] 10.222.0.27 () {54 vars in 927 bytes} [Tue Aug 30 01:42:45 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 01:42:45.680120 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b30a86b4-771b-46f4-8a2a-e799196c2cff None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:45.680589 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b30a86b4-771b-46f4-8a2a-e799196c2cff None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:45.681113 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b30a86b4-771b-46f4-8a2a-e799196c2cff None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:45.691580 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b943a725-97e3-4488-a3d2-86cc67e28352 None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:42:45.718033 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-b30a86b4-771b-46f4-8a2a-e799196c2cff None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 01:42:45.719249 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-b30a86b4-771b-46f4-8a2a-e799196c2cff None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:42:45.753856 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02668c48-1743-473b-962f-5b8a20213770 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:42:45.754506 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02668c48-1743-473b-962f-5b8a20213770 None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:45.754782 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02668c48-1743-473b-962f-5b8a20213770 None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:45.755057 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02668c48-1743-473b-962f-5b8a20213770 None admin] PATH_INFO: `/v3/projects/67913d5682a443069a5eed96f179751f` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:45.757783 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02668c48-1743-473b-962f-5b8a20213770 None admin] RBAC: Authorizing `identity:get_project(project_id=67913d5682a443069a5eed96f179751f)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:42:45.759298 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02668c48-1743-473b-962f-5b8a20213770 None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:42:45.771475 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 147/294] 10.222.0.27 () {56 vars in 1186 bytes} [Tue Aug 30 01:42:45 2022] GET /identity/v3/projects/67913d5682a443069a5eed96f179751f => generated 302 bytes in 226 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:42:45.781245 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02668c48-1743-473b-962f-5b8a20213770 None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:42:45.864251 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 148/295] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Aug 30 01:42:45 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 185 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Aug 30 01:42:45.910426 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:42:45.917347 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02668c48-1743-473b-962f-5b8a20213770 None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:42:45.954080 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d606a49-da8b-4bd2-b622-6ae3ad67d02c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:42:45.954594 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d606a49-da8b-4bd2-b622-6ae3ad67d02c None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:45.954871 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d606a49-da8b-4bd2-b622-6ae3ad67d02c None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:45.955140 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d606a49-da8b-4bd2-b622-6ae3ad67d02c None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:45.955930 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d606a49-da8b-4bd2-b622-6ae3ad67d02c None admin] RBAC: Adding query filter params (role.id=1fb81afcb9f245f3b989add94598a063, scope.project.id=67913d5682a443069a5eed96f179751f, user.id=b3189d44e5c846e88be8df57633d93cd) {{(pid=81578) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:42:45.956237 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d606a49-da8b-4bd2-b622-6ae3ad67d02c None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:42:45.957734 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d606a49-da8b-4bd2-b622-6ae3ad67d02c None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:42:45.964482 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 148/296] 10.222.0.27 () {56 vars in 1374 bytes} [Tue Aug 30 01:42:45 2022] GET /identity/v3/role_assignments?user.id=b3189d44e5c846e88be8df57633d93cd&scope.project.id=67913d5682a443069a5eed96f179751f&role.id=1fb81afcb9f245f3b989add94598a063 => generated 608 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:42:46.236949 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b30a86b4-771b-46f4-8a2a-e799196c2cff None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:42:46.308566 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:42:46.352896 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b30a86b4-771b-46f4-8a2a-e799196c2cff None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:42:46.394598 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a302ef3f-d3d4-434f-b332-aebb7de1bf95 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:42:46.395257 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a302ef3f-d3d4-434f-b332-aebb7de1bf95 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:46.395530 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a302ef3f-d3d4-434f-b332-aebb7de1bf95 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:46.395799 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a302ef3f-d3d4-434f-b332-aebb7de1bf95 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:46.405570 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94755b57-1752-404f-bc24-29d28de98e30 None None] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:46.406155 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94755b57-1752-404f-bc24-29d28de98e30 None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:46.406625 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94755b57-1752-404f-bc24-29d28de98e30 None None] PATH_INFO: `/` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:46.407390 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a302ef3f-d3d4-434f-b332-aebb7de1bf95 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 30 01:42:46.408653 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 149/297] 10.222.0.27 () {54 vars in 927 bytes} [Tue Aug 30 01:42:46 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 01:42:46.412510 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application [None req-a302ef3f-d3d4-434f-b332-aebb7de1bf95 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Aug 30 01:42:46.412510 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 01:42:46.412510 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 01:42:46.412510 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 01:42:46.412510 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 01:42:46.412510 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 01:42:46.412510 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 01:42:46.412510 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 01:42:46.412510 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 01:42:46.412510 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 01:42:46.412510 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 01:42:46.412510 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 01:42:46.412510 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 30 01:42:46.412510 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 30 01:42:46.412510 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 30 01:42:46.412510 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 30 01:42:46.413414 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 30 01:42:46.413414 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 30 01:42:46.413414 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 30 01:42:46.413414 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 30 01:42:46.413414 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 01:42:46.413414 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 01:42:46.413414 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 01:42:46.413414 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 01:42:46.413414 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 01:42:46.413414 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 01:42:46.413414 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 01:42:46.413414 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application with Lock( Aug 30 01:42:46.413414 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 01:42:46.413414 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._enter() Aug 30 01:42:46.413414 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 01:42:46.413414 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 01:42:46.414819 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 01:42:46.414819 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.creator() Aug 30 01:42:46.414819 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 01:42:46.414819 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application created_value = creator( Aug 30 01:42:46.414819 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 30 01:42:46.414819 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 30 01:42:46.414819 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 30 01:42:46.414819 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 30 01:42:46.414819 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 30 01:42:46.414819 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 30 01:42:46.414819 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Aug 30 01:42:46.414819 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Aug 30 01:42:46.414819 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 149/298] 10.222.0.27 () {56 vars in 1099 bytes} [Tue Aug 30 01:42:46 2022] GET /identity/v3/roles/member => generated 84 bytes in 178 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 01:42:46.420009 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94755b57-1752-404f-bc24-29d28de98e30 None None] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:42:46.421175 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-459533e9-9bc4-4606-8c78-60efc2c230e5 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:46.421303 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-459533e9-9bc4-4606-8c78-60efc2c230e5 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:46.421487 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-459533e9-9bc4-4606-8c78-60efc2c230e5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:46.443616 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-459533e9-9bc4-4606-8c78-60efc2c230e5 None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 01:42:46.444499 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-459533e9-9bc4-4606-8c78-60efc2c230e5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:42:46.538879 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 150/299] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Aug 30 01:42:46 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 118 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Aug 30 01:42:46.555160 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94755b57-1752-404f-bc24-29d28de98e30 None None] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:42:46.591507 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c4d2891-02f4-40a0-8265-83fd5efc8540 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:42:46.592238 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c4d2891-02f4-40a0-8265-83fd5efc8540 None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:46.592520 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c4d2891-02f4-40a0-8265-83fd5efc8540 None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:46.592828 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c4d2891-02f4-40a0-8265-83fd5efc8540 None admin] PATH_INFO: `/v3/roles` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:46.593517 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c4d2891-02f4-40a0-8265-83fd5efc8540 None admin] RBAC: Adding query filter params (name=member) {{(pid=81578) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:42:46.593827 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c4d2891-02f4-40a0-8265-83fd5efc8540 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:42:46.595253 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c4d2891-02f4-40a0-8265-83fd5efc8540 None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:42:46.606092 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 150/300] 10.222.0.27 () {56 vars in 1101 bytes} [Tue Aug 30 01:42:46 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 186 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:42:46.612582 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-459533e9-9bc4-4606-8c78-60efc2c230e5 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:42:46.676929 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:42:46.753249 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-459533e9-9bc4-4606-8c78-60efc2c230e5 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:42:46.791634 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5325b062-b38f-4a28-923d-772634cc5d5f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:42:46.792192 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5325b062-b38f-4a28-923d-772634cc5d5f None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:46.792472 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5325b062-b38f-4a28-923d-772634cc5d5f None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:46.792746 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5325b062-b38f-4a28-923d-772634cc5d5f None admin] PATH_INFO: `/v3/users/87b7b50b8f3547969f5ecaaed9ac936d` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:46.793912 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5325b062-b38f-4a28-923d-772634cc5d5f None admin] RBAC: Authorizing `identity:get_user(user_id=87b7b50b8f3547969f5ecaaed9ac936d)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:42:46.795353 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5325b062-b38f-4a28-923d-772634cc5d5f None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:42:46.797254 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 151/301] 10.222.0.27 () {56 vars in 1177 bytes} [Tue Aug 30 01:42:46 2022] GET /identity/v3/users/87b7b50b8f3547969f5ecaaed9ac936d => generated 293 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:42:46.802946 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c4d2891-02f4-40a0-8265-83fd5efc8540 None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:42:46.809085 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa91963b-e224-413d-b1cf-4552c1e6fe56 None None] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:46.809640 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa91963b-e224-413d-b1cf-4552c1e6fe56 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:46.810091 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa91963b-e224-413d-b1cf-4552c1e6fe56 None None] PATH_INFO: `/` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:46.811905 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 152/302] 10.222.0.27 () {54 vars in 927 bytes} [Tue Aug 30 01:42:46 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 01:42:46.818231 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1a0fd7c-14db-4bdb-af1a-159fca9c8fa6 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:46.818537 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1a0fd7c-14db-4bdb-af1a-159fca9c8fa6 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:46.818775 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1a0fd7c-14db-4bdb-af1a-159fca9c8fa6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:46.835483 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:42:46.838209 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-a1a0fd7c-14db-4bdb-af1a-159fca9c8fa6 None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 01:42:46.839433 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-a1a0fd7c-14db-4bdb-af1a-159fca9c8fa6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:42:46.915885 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 153/303] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Aug 30 01:42:46 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Aug 30 01:42:46.916752 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1a0fd7c-14db-4bdb-af1a-159fca9c8fa6 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:42:46.921392 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c4d2891-02f4-40a0-8265-83fd5efc8540 None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:42:46.960476 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df4f11b7-ad4e-4be9-bc6d-66b1ced9bd22 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:42:46.961290 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df4f11b7-ad4e-4be9-bc6d-66b1ced9bd22 None admin] REQUEST_METHOD: `PUT` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:46.961813 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df4f11b7-ad4e-4be9-bc6d-66b1ced9bd22 None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:46.962170 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df4f11b7-ad4e-4be9-bc6d-66b1ced9bd22 None admin] PATH_INFO: `/v3/system/users/87b7b50b8f3547969f5ecaaed9ac936d/roles/1fb81afcb9f245f3b989add94598a063` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:46.963847 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df4f11b7-ad4e-4be9-bc6d-66b1ced9bd22 None admin] RBAC: Authorizing `identity:create_system_grant_for_user(user_id=87b7b50b8f3547969f5ecaaed9ac936d, role_id=1fb81afcb9f245f3b989add94598a063)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:42:46.966190 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df4f11b7-ad4e-4be9-bc6d-66b1ced9bd22 None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:42:46.985913 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:42:47.035441 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1a0fd7c-14db-4bdb-af1a-159fca9c8fa6 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:42:47.069869 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8eb8748b-8d29-483c-a296-ee7b2b5833c1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:42:47.072882 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8eb8748b-8d29-483c-a296-ee7b2b5833c1 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:47.072882 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8eb8748b-8d29-483c-a296-ee7b2b5833c1 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:47.072882 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8eb8748b-8d29-483c-a296-ee7b2b5833c1 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:47.073165 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 151/304] 10.222.0.27 () {58 vars in 1334 bytes} [Tue Aug 30 01:42:46 2022] PUT /identity/v3/system/users/87b7b50b8f3547969f5ecaaed9ac936d/roles/1fb81afcb9f245f3b989add94598a063 => generated 0 bytes in 269 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 30 01:42:47.073693 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81540580-b697-4ff9-a3a7-364090a69adb None None] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:47.073980 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81540580-b697-4ff9-a3a7-364090a69adb None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:47.074249 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81540580-b697-4ff9-a3a7-364090a69adb None None] PATH_INFO: `/` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:47.075698 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 152/305] 10.222.0.27 () {54 vars in 927 bytes} [Tue Aug 30 01:42:47 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 01:42:47.081727 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1673d648-7062-4d09-82ab-3516d1d4231b None None] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:47.082365 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1673d648-7062-4d09-82ab-3516d1d4231b None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:47.082793 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1673d648-7062-4d09-82ab-3516d1d4231b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:47.091454 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8eb8748b-8d29-483c-a296-ee7b2b5833c1 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 30 01:42:47.096212 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application [None req-8eb8748b-8d29-483c-a296-ee7b2b5833c1 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Aug 30 01:42:47.096212 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 01:42:47.096212 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 01:42:47.096212 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 01:42:47.096212 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 01:42:47.096212 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 01:42:47.096212 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 01:42:47.096212 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 01:42:47.096212 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 01:42:47.096212 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 01:42:47.096212 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 01:42:47.096212 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 01:42:47.096212 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 30 01:42:47.096212 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 30 01:42:47.096212 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 30 01:42:47.096212 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 30 01:42:47.097265 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 30 01:42:47.097265 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 30 01:42:47.097265 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 30 01:42:47.097265 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 30 01:42:47.097265 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 01:42:47.097265 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 01:42:47.097265 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 01:42:47.097265 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 01:42:47.097265 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 01:42:47.097265 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 01:42:47.097265 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 01:42:47.097265 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application with Lock( Aug 30 01:42:47.097265 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 01:42:47.097265 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._enter() Aug 30 01:42:47.097265 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 01:42:47.097265 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 01:42:47.098369 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 01:42:47.098369 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.creator() Aug 30 01:42:47.098369 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 01:42:47.098369 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application created_value = creator( Aug 30 01:42:47.098369 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 30 01:42:47.098369 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 30 01:42:47.098369 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 30 01:42:47.098369 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 30 01:42:47.098369 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 30 01:42:47.098369 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 30 01:42:47.098369 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Aug 30 01:42:47.098369 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Aug 30 01:42:47.098369 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 154/306] 10.222.0.27 () {56 vars in 1099 bytes} [Tue Aug 30 01:42:46 2022] GET /identity/v3/roles/reader => generated 84 bytes in 181 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 01:42:47.104427 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8eb8748b-8d29-483c-a296-ee7b2b5833c1 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:42:47.108049 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.auth.core [None req-1673d648-7062-4d09-82ab-3516d1d4231b None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81578) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 01:42:47.109635 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-1673d648-7062-4d09-82ab-3516d1d4231b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:42:47.185249 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 153/307] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Aug 30 01:42:47 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Aug 30 01:42:47.203993 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1673d648-7062-4d09-82ab-3516d1d4231b None None] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:42:47.222186 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:42:47.232183 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8eb8748b-8d29-483c-a296-ee7b2b5833c1 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:42:47.248285 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:42:47.278386 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e647c2b0-8d11-481b-88d8-8df822dba4ab None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:42:47.279546 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e647c2b0-8d11-481b-88d8-8df822dba4ab None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:47.280370 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e647c2b0-8d11-481b-88d8-8df822dba4ab None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:47.281126 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e647c2b0-8d11-481b-88d8-8df822dba4ab None admin] PATH_INFO: `/v3/roles` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:47.282013 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e647c2b0-8d11-481b-88d8-8df822dba4ab None admin] RBAC: Adding query filter params (name=reader) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:42:47.282694 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e647c2b0-8d11-481b-88d8-8df822dba4ab None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:42:47.285037 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e647c2b0-8d11-481b-88d8-8df822dba4ab None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:42:47.298619 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 155/308] 10.222.0.27 () {56 vars in 1101 bytes} [Tue Aug 30 01:42:47 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 194 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 30 01:42:47.303607 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e647c2b0-8d11-481b-88d8-8df822dba4ab None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:42:47.325515 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1673d648-7062-4d09-82ab-3516d1d4231b None None] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:42:47.362307 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11ed84a1-24e9-4732-b878-a70694c3bb66 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:42:47.363107 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11ed84a1-24e9-4732-b878-a70694c3bb66 None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:47.363478 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11ed84a1-24e9-4732-b878-a70694c3bb66 None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:47.363811 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11ed84a1-24e9-4732-b878-a70694c3bb66 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:47.364979 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11ed84a1-24e9-4732-b878-a70694c3bb66 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:42:47.366529 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11ed84a1-24e9-4732-b878-a70694c3bb66 None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:42:47.372709 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 154/309] 10.222.0.27 () {56 vars in 1108 bytes} [Tue Aug 30 01:42:47 2022] GET /identity/v3/domains/default => generated 199 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 30 01:42:47.374178 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11ed84a1-24e9-4732-b878-a70694c3bb66 None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:42:47.436171 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e647c2b0-8d11-481b-88d8-8df822dba4ab None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:42:47.464330 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:42:47.478803 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d6f6a87-3989-463c-a8b0-4366b3a5712e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:42:47.479436 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d6f6a87-3989-463c-a8b0-4366b3a5712e None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:47.479712 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d6f6a87-3989-463c-a8b0-4366b3a5712e None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:47.479981 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d6f6a87-3989-463c-a8b0-4366b3a5712e None admin] PATH_INFO: `/v3/users/7075d18eee8744e8a1a4fd95076896e1` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:47.486207 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11ed84a1-24e9-4732-b878-a70694c3bb66 None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:42:47.494503 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d6f6a87-3989-463c-a8b0-4366b3a5712e None admin] RBAC: Authorizing `identity:get_user(user_id=7075d18eee8744e8a1a4fd95076896e1)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:42:47.495985 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d6f6a87-3989-463c-a8b0-4366b3a5712e None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:42:47.502344 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 156/310] 10.222.0.27 () {56 vars in 1177 bytes} [Tue Aug 30 01:42:47 2022] GET /identity/v3/users/7075d18eee8744e8a1a4fd95076896e1 => generated 293 bytes in 194 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:42:47.502344 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d6f6a87-3989-463c-a8b0-4366b3a5712e None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:42:47.526008 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be737fed-77c2-44eb-9c28-013d3eb33aa6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:42:47.526623 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be737fed-77c2-44eb-9c28-013d3eb33aa6 None admin] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:47.526898 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be737fed-77c2-44eb-9c28-013d3eb33aa6 None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:47.527168 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be737fed-77c2-44eb-9c28-013d3eb33aa6 None admin] PATH_INFO: `/v3/groups` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:47.527825 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be737fed-77c2-44eb-9c28-013d3eb33aa6 None admin] RBAC: Authorizing `identity:create_group()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:42:47.529254 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be737fed-77c2-44eb-9c28-013d3eb33aa6 None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:42:47.536042 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:42:47.604834 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d6f6a87-3989-463c-a8b0-4366b3a5712e None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:42:47.629522 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 155/311] 10.222.0.27 () {60 vars in 1134 bytes} [Tue Aug 30 01:42:47 2022] POST /identity/v3/groups => generated 229 bytes in 246 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 30 01:42:47.629522 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be737fed-77c2-44eb-9c28-013d3eb33aa6 None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:42:47.655759 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b8ae904-f861-4219-98f7-33c16dee5f70 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:42:47.656519 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b8ae904-f861-4219-98f7-33c16dee5f70 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:47.656800 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b8ae904-f861-4219-98f7-33c16dee5f70 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:47.657072 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b8ae904-f861-4219-98f7-33c16dee5f70 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:47.671476 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b8ae904-f861-4219-98f7-33c16dee5f70 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 30 01:42:47.677147 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application [None req-3b8ae904-f861-4219-98f7-33c16dee5f70 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Aug 30 01:42:47.677147 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 01:42:47.677147 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 01:42:47.677147 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 01:42:47.677147 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 01:42:47.677147 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 01:42:47.677147 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 01:42:47.677147 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 01:42:47.677147 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 01:42:47.677147 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 01:42:47.677147 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 01:42:47.677147 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 01:42:47.677147 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 30 01:42:47.677147 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 30 01:42:47.677147 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 30 01:42:47.677147 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 30 01:42:47.678152 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 30 01:42:47.678152 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 30 01:42:47.678152 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 30 01:42:47.678152 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 30 01:42:47.678152 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 01:42:47.678152 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 01:42:47.678152 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 01:42:47.678152 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 01:42:47.678152 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 01:42:47.678152 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 01:42:47.678152 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 01:42:47.678152 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application with Lock( Aug 30 01:42:47.678152 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 01:42:47.678152 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._enter() Aug 30 01:42:47.678152 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 01:42:47.678152 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 01:42:47.679613 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 01:42:47.679613 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.creator() Aug 30 01:42:47.679613 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 01:42:47.679613 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application created_value = creator( Aug 30 01:42:47.679613 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 30 01:42:47.679613 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 30 01:42:47.679613 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 30 01:42:47.679613 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 30 01:42:47.679613 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 30 01:42:47.679613 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 30 01:42:47.679613 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Aug 30 01:42:47.679613 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Aug 30 01:42:47.679613 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 157/312] 10.222.0.27 () {56 vars in 1099 bytes} [Tue Aug 30 01:42:47 2022] GET /identity/v3/roles/reader => generated 84 bytes in 180 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 01:42:47.685169 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b8ae904-f861-4219-98f7-33c16dee5f70 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:42:47.780878 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be737fed-77c2-44eb-9c28-013d3eb33aa6 None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:42:47.836795 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1d62cc3-1306-495d-bd5a-de6b3dbea172 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:42:47.837434 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1d62cc3-1306-495d-bd5a-de6b3dbea172 None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:47.837704 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1d62cc3-1306-495d-bd5a-de6b3dbea172 None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:47.837934 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1d62cc3-1306-495d-bd5a-de6b3dbea172 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:47.838665 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1d62cc3-1306-495d-bd5a-de6b3dbea172 None admin] RBAC: Adding query filter params (role.id=ffde3a55b5dd43149c18c72b7f2fe6ae, user.id=7075d18eee8744e8a1a4fd95076896e1, scope.system=all) {{(pid=81578) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:42:47.838969 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1d62cc3-1306-495d-bd5a-de6b3dbea172 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:42:47.840469 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1d62cc3-1306-495d-bd5a-de6b3dbea172 None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:42:47.849741 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 156/313] 10.222.0.27 () {56 vars in 1308 bytes} [Tue Aug 30 01:42:47 2022] GET /identity/v3/role_assignments?user.id=7075d18eee8744e8a1a4fd95076896e1&scope.system=all&role.id=ffde3a55b5dd43149c18c72b7f2fe6ae => generated 226 bytes in 229 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:42:47.889360 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b8ae904-f861-4219-98f7-33c16dee5f70 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:42:47.891377 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:42:47.931487 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29deaeb8-4f0e-4e11-bf6e-edb37fc1bfa3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:42:47.932153 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29deaeb8-4f0e-4e11-bf6e-edb37fc1bfa3 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:47.932431 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29deaeb8-4f0e-4e11-bf6e-edb37fc1bfa3 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:47.932705 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29deaeb8-4f0e-4e11-bf6e-edb37fc1bfa3 None admin] PATH_INFO: `/v3/roles` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:47.933518 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29deaeb8-4f0e-4e11-bf6e-edb37fc1bfa3 None admin] RBAC: Adding query filter params (name=reader) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:42:47.933833 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29deaeb8-4f0e-4e11-bf6e-edb37fc1bfa3 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:42:47.935340 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29deaeb8-4f0e-4e11-bf6e-edb37fc1bfa3 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:42:47.946997 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 158/314] 10.222.0.27 () {56 vars in 1101 bytes} [Tue Aug 30 01:42:47 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 262 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:42:47.952331 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1d62cc3-1306-495d-bd5a-de6b3dbea172 None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:42:47.964076 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:42:48.072801 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1d62cc3-1306-495d-bd5a-de6b3dbea172 None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:42:48.108692 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d397a27e-7eba-43c0-aad7-d3f0713908bd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:42:48.109336 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d397a27e-7eba-43c0-aad7-d3f0713908bd None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:48.109657 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d397a27e-7eba-43c0-aad7-d3f0713908bd None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:48.109934 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d397a27e-7eba-43c0-aad7-d3f0713908bd None admin] PATH_INFO: `/v3/users/2ed4eca98551441e8a808afce7833f89` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:48.111092 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d397a27e-7eba-43c0-aad7-d3f0713908bd None admin] RBAC: Authorizing `identity:get_user(user_id=2ed4eca98551441e8a808afce7833f89)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:42:48.112617 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d397a27e-7eba-43c0-aad7-d3f0713908bd None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:42:48.114476 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 157/315] 10.222.0.27 () {56 vars in 1177 bytes} [Tue Aug 30 01:42:47 2022] GET /identity/v3/users/2ed4eca98551441e8a808afce7833f89 => generated 297 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:42:48.120843 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29deaeb8-4f0e-4e11-bf6e-edb37fc1bfa3 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:42:48.228283 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29deaeb8-4f0e-4e11-bf6e-edb37fc1bfa3 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:42:48.264137 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca87520b-4e4d-4735-809f-baa6b8ce121a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:42:48.264772 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca87520b-4e4d-4735-809f-baa6b8ce121a None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:48.265039 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca87520b-4e4d-4735-809f-baa6b8ce121a None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:48.265289 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca87520b-4e4d-4735-809f-baa6b8ce121a None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:48.293222 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:42:48.342253 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca87520b-4e4d-4735-809f-baa6b8ce121a None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:42:48.344073 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca87520b-4e4d-4735-809f-baa6b8ce121a None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:42:48.449969 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 159/316] 10.222.0.27 () {58 vars in 1282 bytes} [Tue Aug 30 01:42:48 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 329 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 30 01:42:48.455332 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d397a27e-7eba-43c0-aad7-d3f0713908bd None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:42:48.545499 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:42:48.571045 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d397a27e-7eba-43c0-aad7-d3f0713908bd None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:42:48.606106 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22fb1310-74d7-4659-828e-ad1f9b98823b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:42:48.606740 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22fb1310-74d7-4659-828e-ad1f9b98823b None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:48.607019 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22fb1310-74d7-4659-828e-ad1f9b98823b None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:48.607292 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22fb1310-74d7-4659-828e-ad1f9b98823b None admin] PATH_INFO: `/v3/projects/67913d5682a443069a5eed96f179751f` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:48.608413 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22fb1310-74d7-4659-828e-ad1f9b98823b None admin] RBAC: Authorizing `identity:get_project(project_id=67913d5682a443069a5eed96f179751f)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:42:48.609897 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22fb1310-74d7-4659-828e-ad1f9b98823b None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:42:48.611822 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 158/317] 10.222.0.27 () {56 vars in 1186 bytes} [Tue Aug 30 01:42:48 2022] GET /identity/v3/projects/67913d5682a443069a5eed96f179751f => generated 302 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:42:48.618523 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca87520b-4e4d-4735-809f-baa6b8ce121a None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:42:48.679038 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:42:48.731123 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca87520b-4e4d-4735-809f-baa6b8ce121a None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:42:48.768178 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-679100ad-91f8-4eb1-a0bd-1ccefcc8b939 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:42:48.768798 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-679100ad-91f8-4eb1-a0bd-1ccefcc8b939 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:48.769085 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-679100ad-91f8-4eb1-a0bd-1ccefcc8b939 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:48.769359 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-679100ad-91f8-4eb1-a0bd-1ccefcc8b939 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:48.770122 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-679100ad-91f8-4eb1-a0bd-1ccefcc8b939 None admin] RBAC: Adding query filter params (role.id=ffde3a55b5dd43149c18c72b7f2fe6ae, scope.project.id=67913d5682a443069a5eed96f179751f, user.id=2ed4eca98551441e8a808afce7833f89) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:42:48.770434 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-679100ad-91f8-4eb1-a0bd-1ccefcc8b939 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:42:48.771882 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-679100ad-91f8-4eb1-a0bd-1ccefcc8b939 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:42:48.778426 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 160/318] 10.222.0.27 () {56 vars in 1374 bytes} [Tue Aug 30 01:42:48 2022] GET /identity/v3/role_assignments?user.id=2ed4eca98551441e8a808afce7833f89&scope.project.id=67913d5682a443069a5eed96f179751f&role.id=ffde3a55b5dd43149c18c72b7f2fe6ae => generated 608 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:42:49.143876 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ad97833-4ab8-4290-8783-fd7a4b036029 None None] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:49.144437 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ad97833-4ab8-4290-8783-fd7a4b036029 None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:49.144437 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ad97833-4ab8-4290-8783-fd7a4b036029 None None] PATH_INFO: `/` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:49.145929 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 159/319] 10.222.0.27 () {54 vars in 927 bytes} [Tue Aug 30 01:42:49 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 01:42:49.151300 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebdc7143-c2b5-4656-b556-06262d436cbd None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:49.151591 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebdc7143-c2b5-4656-b556-06262d436cbd None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:49.151870 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebdc7143-c2b5-4656-b556-06262d436cbd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:49.169931 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-ebdc7143-c2b5-4656-b556-06262d436cbd None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 01:42:49.171094 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-ebdc7143-c2b5-4656-b556-06262d436cbd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:42:49.263508 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 161/320] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Aug 30 01:42:49 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 113 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 30 01:42:49.535848 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ad97833-4ab8-4290-8783-fd7a4b036029 None None] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:42:49.639433 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ad97833-4ab8-4290-8783-fd7a4b036029 None None] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:42:49.673432 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09759f31-01ba-4a5d-bf01-045d2d19015e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:42:49.674097 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09759f31-01ba-4a5d-bf01-045d2d19015e None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:49.674428 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09759f31-01ba-4a5d-bf01-045d2d19015e None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:49.674697 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09759f31-01ba-4a5d-bf01-045d2d19015e None admin] PATH_INFO: `/v3/roles/member` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:49.686097 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09759f31-01ba-4a5d-bf01-045d2d19015e None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 30 01:42:49.691362 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application [None req-09759f31-01ba-4a5d-bf01-045d2d19015e None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Aug 30 01:42:49.691362 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 01:42:49.691362 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 01:42:49.691362 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 01:42:49.691362 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 01:42:49.691362 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 01:42:49.691362 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 01:42:49.691362 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 01:42:49.691362 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 01:42:49.691362 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 01:42:49.691362 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 01:42:49.691362 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 01:42:49.691362 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 30 01:42:49.691362 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 30 01:42:49.691362 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 30 01:42:49.691362 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 30 01:42:49.692255 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 30 01:42:49.692255 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 30 01:42:49.692255 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 30 01:42:49.692255 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 30 01:42:49.692255 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 01:42:49.692255 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 01:42:49.692255 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 01:42:49.692255 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 01:42:49.692255 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 01:42:49.692255 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 01:42:49.692255 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 01:42:49.692255 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application with Lock( Aug 30 01:42:49.692255 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 01:42:49.692255 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._enter() Aug 30 01:42:49.692255 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 01:42:49.692255 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 01:42:49.693044 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 01:42:49.693044 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.creator() Aug 30 01:42:49.693044 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 01:42:49.693044 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application created_value = creator( Aug 30 01:42:49.693044 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 30 01:42:49.693044 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 30 01:42:49.693044 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 30 01:42:49.693044 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 30 01:42:49.693044 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 30 01:42:49.693044 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 30 01:42:49.693044 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Aug 30 01:42:49.693044 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application Aug 30 01:42:49.697422 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 160/321] 10.222.0.27 () {56 vars in 1099 bytes} [Tue Aug 30 01:42:49 2022] GET /identity/v3/roles/member => generated 84 bytes in 158 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 01:42:49.698823 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebdc7143-c2b5-4656-b556-06262d436cbd None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:42:49.759239 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:42:49.778701 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-516f2897-82da-4d23-9cd6-152263aa915e None None] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:49.779008 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-516f2897-82da-4d23-9cd6-152263aa915e None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:49.779514 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-516f2897-82da-4d23-9cd6-152263aa915e None None] PATH_INFO: `/` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:49.781174 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 161/322] 10.222.0.27 () {54 vars in 927 bytes} [Tue Aug 30 01:42:49 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 30 01:42:49.786322 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bb12a66-5016-47b7-946b-5706473f52f2 None None] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:49.786537 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bb12a66-5016-47b7-946b-5706473f52f2 None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:49.786804 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bb12a66-5016-47b7-946b-5706473f52f2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:49.793737 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:42:49.805988 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.auth.core [None req-1bb12a66-5016-47b7-946b-5706473f52f2 None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81578) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 01:42:49.807185 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-1bb12a66-5016-47b7-946b-5706473f52f2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:42:49.826249 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebdc7143-c2b5-4656-b556-06262d436cbd None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:42:49.861465 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b826f5b-4c73-45d0-8c0e-96bf58619770 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:42:49.862091 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b826f5b-4c73-45d0-8c0e-96bf58619770 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:49.862361 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b826f5b-4c73-45d0-8c0e-96bf58619770 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:49.862639 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b826f5b-4c73-45d0-8c0e-96bf58619770 None admin] PATH_INFO: `/v3/roles` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:49.863302 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b826f5b-4c73-45d0-8c0e-96bf58619770 None admin] RBAC: Adding query filter params (name=member) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:42:49.863602 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b826f5b-4c73-45d0-8c0e-96bf58619770 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:42:49.865004 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b826f5b-4c73-45d0-8c0e-96bf58619770 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:42:49.874013 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 162/323] 10.222.0.27 () {56 vars in 1101 bytes} [Tue Aug 30 01:42:49 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:42:49.875229 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e29ef98-6c96-4d3f-b496-8873c3632682 None None] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:49.875517 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e29ef98-6c96-4d3f-b496-8873c3632682 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:49.875792 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e29ef98-6c96-4d3f-b496-8873c3632682 None None] PATH_INFO: `/` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:49.877012 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 163/324] 10.222.0.27 () {54 vars in 927 bytes} [Tue Aug 30 01:42:49 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 01:42:49.879532 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 162/325] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Aug 30 01:42:49 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Aug 30 01:42:49.881842 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e29ef98-6c96-4d3f-b496-8873c3632682 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:42:49.882334 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eeb81136-0a1b-411d-8680-c67625494213 None None] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:49.882676 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eeb81136-0a1b-411d-8680-c67625494213 None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:49.882877 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eeb81136-0a1b-411d-8680-c67625494213 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:49.901525 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.auth.core [None req-eeb81136-0a1b-411d-8680-c67625494213 None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81578) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 01:42:49.902702 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-eeb81136-0a1b-411d-8680-c67625494213 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:42:49.979311 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 163/326] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Aug 30 01:42:49 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 30 01:42:50.001411 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e29ef98-6c96-4d3f-b496-8873c3632682 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:42:50.013586 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:42:50.037750 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b96e9893-0e5a-486d-ba18-880792d7d28a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:42:50.038303 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b96e9893-0e5a-486d-ba18-880792d7d28a None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:50.038579 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b96e9893-0e5a-486d-ba18-880792d7d28a None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:50.038897 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b96e9893-0e5a-486d-ba18-880792d7d28a None admin] PATH_INFO: `/v3/users/87b7b50b8f3547969f5ecaaed9ac936d` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:50.040757 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b96e9893-0e5a-486d-ba18-880792d7d28a None admin] RBAC: Authorizing `identity:get_user(user_id=87b7b50b8f3547969f5ecaaed9ac936d)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:42:50.042431 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b96e9893-0e5a-486d-ba18-880792d7d28a None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:42:50.049321 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 164/327] 10.222.0.27 () {56 vars in 1177 bytes} [Tue Aug 30 01:42:49 2022] GET /identity/v3/users/87b7b50b8f3547969f5ecaaed9ac936d => generated 293 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:42:50.051276 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eeb81136-0a1b-411d-8680-c67625494213 None None] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:42:50.115821 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:42:50.156406 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b96e9893-0e5a-486d-ba18-880792d7d28a None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:42:50.162484 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eeb81136-0a1b-411d-8680-c67625494213 None None] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:42:50.207504 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97763256-c0e2-4b50-8f95-4fe892bb6f29 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:42:50.208108 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97763256-c0e2-4b50-8f95-4fe892bb6f29 None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:50.208384 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97763256-c0e2-4b50-8f95-4fe892bb6f29 None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:50.208654 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97763256-c0e2-4b50-8f95-4fe892bb6f29 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:50.209359 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97763256-c0e2-4b50-8f95-4fe892bb6f29 None admin] RBAC: Adding query filter params (role.id=1fb81afcb9f245f3b989add94598a063, user.id=87b7b50b8f3547969f5ecaaed9ac936d, scope.system=all) {{(pid=81578) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:42:50.209682 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97763256-c0e2-4b50-8f95-4fe892bb6f29 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:42:50.211178 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97763256-c0e2-4b50-8f95-4fe892bb6f29 None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:42:50.223932 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 164/328] 10.222.0.27 () {56 vars in 1308 bytes} [Tue Aug 30 01:42:50 2022] GET /identity/v3/role_assignments?user.id=87b7b50b8f3547969f5ecaaed9ac936d&scope.system=all&role.id=1fb81afcb9f245f3b989add94598a063 => generated 510 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:42:50.254452 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97763256-c0e2-4b50-8f95-4fe892bb6f29 None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:42:50.281622 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b96e9893-0e5a-486d-ba18-880792d7d28a None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:42:50.322129 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7eba9acc-6875-4b50-bae5-4893511babb4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:42:50.322779 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7eba9acc-6875-4b50-bae5-4893511babb4 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:50.323057 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7eba9acc-6875-4b50-bae5-4893511babb4 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:50.323326 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7eba9acc-6875-4b50-bae5-4893511babb4 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:50.324491 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7eba9acc-6875-4b50-bae5-4893511babb4 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:42:50.325994 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7eba9acc-6875-4b50-bae5-4893511babb4 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:42:50.327999 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 165/329] 10.222.0.27 () {56 vars in 1108 bytes} [Tue Aug 30 01:42:50 2022] GET /identity/v3/domains/default => generated 199 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:42:50.333859 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7eba9acc-6875-4b50-bae5-4893511babb4 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:42:50.385159 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97763256-c0e2-4b50-8f95-4fe892bb6f29 None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:42:50.401600 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:42:50.409396 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:42:50.426408 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1288626-f3fd-4dc5-bb9e-edba043829a1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:42:50.427045 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1288626-f3fd-4dc5-bb9e-edba043829a1 None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:50.427319 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1288626-f3fd-4dc5-bb9e-edba043829a1 None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:50.427588 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1288626-f3fd-4dc5-bb9e-edba043829a1 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:50.440552 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1288626-f3fd-4dc5-bb9e-edba043829a1 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 30 01:42:50.445441 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application [None req-e1288626-f3fd-4dc5-bb9e-edba043829a1 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Aug 30 01:42:50.445441 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 01:42:50.445441 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 01:42:50.445441 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 01:42:50.445441 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 01:42:50.445441 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 01:42:50.445441 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 01:42:50.445441 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 01:42:50.445441 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 01:42:50.445441 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 01:42:50.445441 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 01:42:50.445441 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 01:42:50.445441 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 30 01:42:50.445441 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 30 01:42:50.445441 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 30 01:42:50.445441 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 30 01:42:50.447654 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 30 01:42:50.447654 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 30 01:42:50.447654 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 30 01:42:50.447654 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 30 01:42:50.447654 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 01:42:50.447654 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 01:42:50.447654 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 01:42:50.447654 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 01:42:50.447654 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 01:42:50.447654 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 01:42:50.447654 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 01:42:50.447654 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application with Lock( Aug 30 01:42:50.447654 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 01:42:50.447654 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._enter() Aug 30 01:42:50.447654 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 01:42:50.447654 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 01:42:50.448614 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 01:42:50.448614 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.creator() Aug 30 01:42:50.448614 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 01:42:50.448614 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application created_value = creator( Aug 30 01:42:50.448614 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 30 01:42:50.448614 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 30 01:42:50.448614 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 30 01:42:50.448614 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 30 01:42:50.448614 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 30 01:42:50.448614 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 30 01:42:50.448614 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Aug 30 01:42:50.448614 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application Aug 30 01:42:50.448614 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 165/330] 10.222.0.27 () {56 vars in 1099 bytes} [Tue Aug 30 01:42:50 2022] GET /identity/v3/roles/reader => generated 84 bytes in 193 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 01:42:50.452826 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1288626-f3fd-4dc5-bb9e-edba043829a1 None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:42:50.462047 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7eba9acc-6875-4b50-bae5-4893511babb4 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:42:50.496054 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86b7c187-a4f4-43ab-a72a-6b0a07df2241 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:42:50.496660 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86b7c187-a4f4-43ab-a72a-6b0a07df2241 None admin] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:50.496939 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86b7c187-a4f4-43ab-a72a-6b0a07df2241 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:50.497208 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86b7c187-a4f4-43ab-a72a-6b0a07df2241 None admin] PATH_INFO: `/v3/groups` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:50.497872 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86b7c187-a4f4-43ab-a72a-6b0a07df2241 None admin] RBAC: Authorizing `identity:create_group()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:42:50.499293 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86b7c187-a4f4-43ab-a72a-6b0a07df2241 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:42:50.562937 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1288626-f3fd-4dc5-bb9e-edba043829a1 None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:42:50.595680 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76c86c7c-336f-44d0-a94d-0b5d52726b24 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:42:50.596247 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76c86c7c-336f-44d0-a94d-0b5d52726b24 None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:50.596520 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76c86c7c-336f-44d0-a94d-0b5d52726b24 None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:50.596785 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76c86c7c-336f-44d0-a94d-0b5d52726b24 None admin] PATH_INFO: `/v3/roles` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:50.597438 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76c86c7c-336f-44d0-a94d-0b5d52726b24 None admin] RBAC: Adding query filter params (name=reader) {{(pid=81578) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:42:50.597767 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76c86c7c-336f-44d0-a94d-0b5d52726b24 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:42:50.599133 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76c86c7c-336f-44d0-a94d-0b5d52726b24 None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:42:50.607397 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 166/331] 10.222.0.27 () {56 vars in 1101 bytes} [Tue Aug 30 01:42:50 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 30 01:42:50.613074 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76c86c7c-336f-44d0-a94d-0b5d52726b24 None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:42:50.625606 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 166/332] 10.222.0.27 () {60 vars in 1134 bytes} [Tue Aug 30 01:42:50 2022] POST /identity/v3/groups => generated 226 bytes in 292 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 30 01:42:50.662189 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:42:50.726514 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76c86c7c-336f-44d0-a94d-0b5d52726b24 None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:42:50.763178 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-291a2fc3-a093-43c5-a978-dad253171d27 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:42:50.763810 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-291a2fc3-a093-43c5-a978-dad253171d27 None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:50.764086 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-291a2fc3-a093-43c5-a978-dad253171d27 None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:50.764353 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-291a2fc3-a093-43c5-a978-dad253171d27 None admin] PATH_INFO: `/v3/users/7075d18eee8744e8a1a4fd95076896e1` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:50.765496 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-291a2fc3-a093-43c5-a978-dad253171d27 None admin] RBAC: Authorizing `identity:get_user(user_id=7075d18eee8744e8a1a4fd95076896e1)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:42:50.767076 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-291a2fc3-a093-43c5-a978-dad253171d27 None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:42:50.768645 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 167/333] 10.222.0.27 () {56 vars in 1177 bytes} [Tue Aug 30 01:42:50 2022] GET /identity/v3/users/7075d18eee8744e8a1a4fd95076896e1 => generated 293 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 30 01:42:50.775478 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86b7c187-a4f4-43ab-a72a-6b0a07df2241 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:42:50.881777 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86b7c187-a4f4-43ab-a72a-6b0a07df2241 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:42:50.889223 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:42:50.919123 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4fbee0e-e75a-41fd-85cc-2b31e3117c98 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:42:50.919783 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4fbee0e-e75a-41fd-85cc-2b31e3117c98 None admin] REQUEST_METHOD: `PUT` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:50.920066 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4fbee0e-e75a-41fd-85cc-2b31e3117c98 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:50.920366 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4fbee0e-e75a-41fd-85cc-2b31e3117c98 None admin] PATH_INFO: `/v3/system/users/7075d18eee8744e8a1a4fd95076896e1/roles/ffde3a55b5dd43149c18c72b7f2fe6ae` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:50.921779 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4fbee0e-e75a-41fd-85cc-2b31e3117c98 None admin] RBAC: Authorizing `identity:create_system_grant_for_user(user_id=7075d18eee8744e8a1a4fd95076896e1, role_id=ffde3a55b5dd43149c18c72b7f2fe6ae)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:42:50.923178 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4fbee0e-e75a-41fd-85cc-2b31e3117c98 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:42:51.020452 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 167/334] 10.222.0.27 () {58 vars in 1334 bytes} [Tue Aug 30 01:42:50 2022] PUT /identity/v3/system/users/7075d18eee8744e8a1a4fd95076896e1/roles/ffde3a55b5dd43149c18c72b7f2fe6ae => generated 0 bytes in 246 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 30 01:42:53.300081 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65209140-e425-4770-9923-2d09f05eee00 None None] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:53.301106 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65209140-e425-4770-9923-2d09f05eee00 None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:53.301620 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65209140-e425-4770-9923-2d09f05eee00 None None] PATH_INFO: `/` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:53.303423 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 168/335] 10.222.0.27 () {54 vars in 927 bytes} [Tue Aug 30 01:42:53 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 01:42:53.309313 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-678ee133-d83a-41f5-8b91-ca86163e6cb2 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:53.310186 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-678ee133-d83a-41f5-8b91-ca86163e6cb2 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:53.310661 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-678ee133-d83a-41f5-8b91-ca86163e6cb2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:53.338552 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-678ee133-d83a-41f5-8b91-ca86163e6cb2 None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 01:42:53.340123 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-678ee133-d83a-41f5-8b91-ca86163e6cb2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:42:53.426165 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 168/336] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Aug 30 01:42:53 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 118 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Aug 30 01:42:53.442959 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20a18413-53de-4406-a1d7-e332ab709a2f None None] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:53.443551 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20a18413-53de-4406-a1d7-e332ab709a2f None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:53.444017 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20a18413-53de-4406-a1d7-e332ab709a2f None None] PATH_INFO: `/` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:53.446598 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 169/337] 10.222.0.27 () {54 vars in 927 bytes} [Tue Aug 30 01:42:53 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 01:42:53.451557 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-088ef754-389b-43e9-ba57-1542493c4014 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:53.451846 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-088ef754-389b-43e9-ba57-1542493c4014 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:53.452128 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-088ef754-389b-43e9-ba57-1542493c4014 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:53.480405 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-088ef754-389b-43e9-ba57-1542493c4014 None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 01:42:53.481598 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-088ef754-389b-43e9-ba57-1542493c4014 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:42:53.563644 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 169/338] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Aug 30 01:42:53 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 113 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 30 01:42:53.606670 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0111790-ada1-475d-948b-01f4139948e2 None None] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:53.607325 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0111790-ada1-475d-948b-01f4139948e2 None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:53.607801 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0111790-ada1-475d-948b-01f4139948e2 None None] PATH_INFO: `/` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:53.609646 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 170/339] 10.222.0.27 () {54 vars in 927 bytes} [Tue Aug 30 01:42:53 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 01:42:53.614596 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1e242ad-1fac-4580-a2f1-b51413073a52 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:53.614885 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1e242ad-1fac-4580-a2f1-b51413073a52 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:53.615160 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1e242ad-1fac-4580-a2f1-b51413073a52 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:53.636694 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-e1e242ad-1fac-4580-a2f1-b51413073a52 None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 01:42:53.637914 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-e1e242ad-1fac-4580-a2f1-b51413073a52 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:42:53.670743 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:42:53.722819 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 170/340] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Aug 30 01:42:53 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 30 01:42:53.758979 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0111790-ada1-475d-948b-01f4139948e2 None None] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:42:53.865917 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a1eb8b1-aba8-41b7-8818-e797a68ad090 None None] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:53.866522 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a1eb8b1-aba8-41b7-8818-e797a68ad090 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:53.866990 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a1eb8b1-aba8-41b7-8818-e797a68ad090 None None] PATH_INFO: `/` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:53.869378 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 171/341] 10.222.0.27 () {54 vars in 927 bytes} [Tue Aug 30 01:42:53 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 01:42:53.877277 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5c45bbc-9c25-4310-b6bc-b87151b55181 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:53.877277 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5c45bbc-9c25-4310-b6bc-b87151b55181 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:53.877277 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5c45bbc-9c25-4310-b6bc-b87151b55181 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:53.912991 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0111790-ada1-475d-948b-01f4139948e2 None None] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:42:53.920076 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-f5c45bbc-9c25-4310-b6bc-b87151b55181 None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 01:42:53.922332 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-f5c45bbc-9c25-4310-b6bc-b87151b55181 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:42:53.941813 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:42:54.000884 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17e2d8b9-a8e1-4b3b-bf5d-2505d536dcce None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:42:54.001893 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17e2d8b9-a8e1-4b3b-bf5d-2505d536dcce None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:54.002463 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17e2d8b9-a8e1-4b3b-bf5d-2505d536dcce None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:54.002966 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17e2d8b9-a8e1-4b3b-bf5d-2505d536dcce None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:54.019440 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17e2d8b9-a8e1-4b3b-bf5d-2505d536dcce None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 30 01:42:54.025094 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application [None req-17e2d8b9-a8e1-4b3b-bf5d-2505d536dcce None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Aug 30 01:42:54.025094 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 01:42:54.025094 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 01:42:54.025094 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 01:42:54.025094 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 01:42:54.025094 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 01:42:54.025094 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 01:42:54.025094 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 01:42:54.025094 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 01:42:54.025094 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 01:42:54.025094 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 01:42:54.025094 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 01:42:54.025094 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 30 01:42:54.025094 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 30 01:42:54.025094 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 30 01:42:54.025094 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 30 01:42:54.026031 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 30 01:42:54.026031 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 30 01:42:54.026031 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 30 01:42:54.026031 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 30 01:42:54.026031 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 01:42:54.026031 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 01:42:54.026031 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 01:42:54.026031 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 01:42:54.026031 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 01:42:54.026031 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 01:42:54.026031 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 01:42:54.026031 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application with Lock( Aug 30 01:42:54.026031 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 01:42:54.026031 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._enter() Aug 30 01:42:54.026031 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 01:42:54.026031 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 01:42:54.026818 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 01:42:54.026818 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.creator() Aug 30 01:42:54.026818 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 01:42:54.026818 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application created_value = creator( Aug 30 01:42:54.026818 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 30 01:42:54.026818 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 30 01:42:54.026818 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 30 01:42:54.026818 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 30 01:42:54.026818 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 30 01:42:54.026818 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 30 01:42:54.026818 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Aug 30 01:42:54.026818 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application Aug 30 01:42:54.031141 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 171/342] 10.222.0.27 () {56 vars in 1114 bytes} [Tue Aug 30 01:42:53 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 269 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 01:42:54.032029 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17e2d8b9-a8e1-4b3b-bf5d-2505d536dcce None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:42:54.094773 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 172/343] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Aug 30 01:42:53 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 220 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Aug 30 01:42:54.095570 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5c45bbc-9c25-4310-b6bc-b87151b55181 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:42:54.135877 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17e2d8b9-a8e1-4b3b-bf5d-2505d536dcce None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:42:54.202241 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57679b5e-cc8b-49d8-9284-51d601d8fe3a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:42:54.202891 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57679b5e-cc8b-49d8-9284-51d601d8fe3a None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:54.203169 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57679b5e-cc8b-49d8-9284-51d601d8fe3a None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:54.203440 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57679b5e-cc8b-49d8-9284-51d601d8fe3a None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:54.224830 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57679b5e-cc8b-49d8-9284-51d601d8fe3a None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 30 01:42:54.233149 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application [None req-57679b5e-cc8b-49d8-9284-51d601d8fe3a None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Aug 30 01:42:54.233149 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 01:42:54.233149 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 01:42:54.233149 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 01:42:54.233149 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 01:42:54.233149 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 01:42:54.233149 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 01:42:54.233149 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 01:42:54.233149 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 01:42:54.233149 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 01:42:54.233149 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 01:42:54.233149 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 01:42:54.233149 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 30 01:42:54.233149 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 30 01:42:54.233149 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 30 01:42:54.233149 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 30 01:42:54.234142 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 30 01:42:54.234142 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 30 01:42:54.234142 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 30 01:42:54.234142 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 30 01:42:54.234142 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 01:42:54.234142 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 01:42:54.234142 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 01:42:54.234142 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 01:42:54.234142 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 01:42:54.234142 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 01:42:54.234142 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 01:42:54.234142 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application with Lock( Aug 30 01:42:54.234142 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 01:42:54.234142 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._enter() Aug 30 01:42:54.234142 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 01:42:54.234142 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 01:42:54.235243 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 01:42:54.235243 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.creator() Aug 30 01:42:54.235243 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 01:42:54.235243 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application created_value = creator( Aug 30 01:42:54.235243 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 30 01:42:54.235243 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 30 01:42:54.235243 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 30 01:42:54.235243 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 30 01:42:54.235243 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 30 01:42:54.235243 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 30 01:42:54.235243 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Aug 30 01:42:54.235243 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application Aug 30 01:42:54.237207 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 172/344] 10.222.0.27 () {56 vars in 1099 bytes} [Tue Aug 30 01:42:54 2022] GET /identity/v3/roles/reader => generated 84 bytes in 204 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 01:42:54.241230 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57679b5e-cc8b-49d8-9284-51d601d8fe3a None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:42:54.269823 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:42:54.283260 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:42:54.299537 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5c45bbc-9c25-4310-b6bc-b87151b55181 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:42:54.368451 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57679b5e-cc8b-49d8-9284-51d601d8fe3a None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:42:54.381961 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddde842e-5077-4940-abfe-dd0c71c97f8e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:42:54.382822 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddde842e-5077-4940-abfe-dd0c71c97f8e None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:54.385490 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddde842e-5077-4940-abfe-dd0c71c97f8e None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:54.385490 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddde842e-5077-4940-abfe-dd0c71c97f8e None admin] PATH_INFO: `/v3/roles` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:54.385490 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ddde842e-5077-4940-abfe-dd0c71c97f8e None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:42:54.385490 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ddde842e-5077-4940-abfe-dd0c71c97f8e None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:42:54.386228 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ddde842e-5077-4940-abfe-dd0c71c97f8e None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:42:54.404520 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 173/345] 10.222.0.27 () {56 vars in 1111 bytes} [Tue Aug 30 01:42:54 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 309 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:42:54.405384 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddde842e-5077-4940-abfe-dd0c71c97f8e None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:42:54.421750 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45914ed8-4603-45ca-985a-02b4736436ce None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:42:54.422436 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45914ed8-4603-45ca-985a-02b4736436ce None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:54.422710 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45914ed8-4603-45ca-985a-02b4736436ce None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:54.422978 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45914ed8-4603-45ca-985a-02b4736436ce None admin] PATH_INFO: `/v3/roles/member` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:54.447749 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45914ed8-4603-45ca-985a-02b4736436ce None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 30 01:42:54.452120 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application [None req-45914ed8-4603-45ca-985a-02b4736436ce None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Aug 30 01:42:54.452120 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 01:42:54.452120 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 01:42:54.452120 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 01:42:54.452120 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 01:42:54.452120 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 01:42:54.452120 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 01:42:54.452120 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 01:42:54.452120 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 01:42:54.452120 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 01:42:54.452120 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 01:42:54.452120 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 01:42:54.452120 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 30 01:42:54.452120 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 30 01:42:54.452120 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 30 01:42:54.452120 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 30 01:42:54.453026 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 30 01:42:54.453026 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 30 01:42:54.453026 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 30 01:42:54.453026 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 30 01:42:54.453026 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 01:42:54.453026 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 01:42:54.453026 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 01:42:54.453026 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 01:42:54.453026 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 01:42:54.453026 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 01:42:54.453026 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 01:42:54.453026 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application with Lock( Aug 30 01:42:54.453026 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 01:42:54.453026 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._enter() Aug 30 01:42:54.453026 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 01:42:54.453026 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 01:42:54.453982 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 01:42:54.453982 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.creator() Aug 30 01:42:54.453982 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 01:42:54.453982 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application created_value = creator( Aug 30 01:42:54.453982 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 30 01:42:54.453982 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 30 01:42:54.453982 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 30 01:42:54.453982 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 30 01:42:54.453982 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 30 01:42:54.453982 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 30 01:42:54.453982 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Aug 30 01:42:54.453982 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application Aug 30 01:42:54.456599 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 173/346] 10.222.0.27 () {56 vars in 1099 bytes} [Tue Aug 30 01:42:54 2022] GET /identity/v3/roles/member => generated 84 bytes in 220 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 01:42:54.458589 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45914ed8-4603-45ca-985a-02b4736436ce None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:42:54.547643 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddde842e-5077-4940-abfe-dd0c71c97f8e None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:42:54.563949 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:42:54.574007 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45914ed8-4603-45ca-985a-02b4736436ce None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:42:54.592798 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b91ada19-4dc0-4303-9662-547f4ef58610 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:42:54.592798 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b91ada19-4dc0-4303-9662-547f4ef58610 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:54.595843 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b91ada19-4dc0-4303-9662-547f4ef58610 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:54.595843 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b91ada19-4dc0-4303-9662-547f4ef58610 None admin] PATH_INFO: `/v3/roles` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:54.595843 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b91ada19-4dc0-4303-9662-547f4ef58610 None admin] RBAC: Adding query filter params (name=reader) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:42:54.595843 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b91ada19-4dc0-4303-9662-547f4ef58610 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:42:54.595843 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b91ada19-4dc0-4303-9662-547f4ef58610 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:42:54.605132 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 174/347] 10.222.0.27 () {56 vars in 1101 bytes} [Tue Aug 30 01:42:54 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 200 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:42:54.606081 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b91ada19-4dc0-4303-9662-547f4ef58610 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:42:54.618656 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1090ace6-d684-4b20-a7d9-29bc61abdc2b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:42:54.619666 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1090ace6-d684-4b20-a7d9-29bc61abdc2b None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:54.619953 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1090ace6-d684-4b20-a7d9-29bc61abdc2b None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:54.620294 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1090ace6-d684-4b20-a7d9-29bc61abdc2b None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:54.625023 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:42:54.712799 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1090ace6-d684-4b20-a7d9-29bc61abdc2b None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:42:54.714397 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1090ace6-d684-4b20-a7d9-29bc61abdc2b None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:42:54.753626 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b91ada19-4dc0-4303-9662-547f4ef58610 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:42:54.808800 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93448c15-75d0-4f89-aa48-851a7d55f744 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:42:54.810329 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93448c15-75d0-4f89-aa48-851a7d55f744 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:54.810951 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93448c15-75d0-4f89-aa48-851a7d55f744 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:54.811919 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93448c15-75d0-4f89-aa48-851a7d55f744 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:54.826021 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93448c15-75d0-4f89-aa48-851a7d55f744 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 30 01:42:54.831789 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application [None req-93448c15-75d0-4f89-aa48-851a7d55f744 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Aug 30 01:42:54.831789 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 01:42:54.831789 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 01:42:54.831789 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 01:42:54.831789 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 01:42:54.831789 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 01:42:54.831789 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 01:42:54.831789 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 01:42:54.831789 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 01:42:54.831789 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 01:42:54.831789 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 01:42:54.831789 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 01:42:54.831789 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 30 01:42:54.831789 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 30 01:42:54.831789 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 30 01:42:54.831789 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 30 01:42:54.832732 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 30 01:42:54.832732 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 30 01:42:54.832732 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 30 01:42:54.832732 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 30 01:42:54.832732 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 01:42:54.832732 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 01:42:54.832732 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 01:42:54.832732 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 01:42:54.832732 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 01:42:54.832732 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 01:42:54.832732 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 01:42:54.832732 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application with Lock( Aug 30 01:42:54.832732 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 01:42:54.832732 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._enter() Aug 30 01:42:54.832732 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 01:42:54.832732 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 01:42:54.833641 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 01:42:54.833641 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.creator() Aug 30 01:42:54.833641 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 01:42:54.833641 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application created_value = creator( Aug 30 01:42:54.833641 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 30 01:42:54.833641 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 30 01:42:54.833641 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 30 01:42:54.833641 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 30 01:42:54.833641 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 30 01:42:54.833641 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 30 01:42:54.833641 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Aug 30 01:42:54.833641 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Aug 30 01:42:54.833641 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 175/348] 10.222.0.27 () {56 vars in 1099 bytes} [Tue Aug 30 01:42:54 2022] GET /identity/v3/roles/member => generated 84 bytes in 227 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 01:42:54.834308 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93448c15-75d0-4f89-aa48-851a7d55f744 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:42:54.848825 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:42:54.862414 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 174/349] 10.222.0.27 () {58 vars in 1282 bytes} [Tue Aug 30 01:42:54 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 405 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 30 01:42:54.863579 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63331ab3-2ec7-4282-84a7-88fc4b7ab0c4 None None] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:54.863856 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63331ab3-2ec7-4282-84a7-88fc4b7ab0c4 None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:54.868372 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63331ab3-2ec7-4282-84a7-88fc4b7ab0c4 None None] PATH_INFO: `/` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:54.868372 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 175/350] 10.222.0.27 () {54 vars in 927 bytes} [Tue Aug 30 01:42:54 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 01:42:54.868372 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63331ab3-2ec7-4282-84a7-88fc4b7ab0c4 None None] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:42:54.940736 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:42:54.979876 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93448c15-75d0-4f89-aa48-851a7d55f744 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:42:55.000094 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63331ab3-2ec7-4282-84a7-88fc4b7ab0c4 None None] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:42:55.036697 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe47cb31-d165-4927-9ca8-a335544fb1b3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:42:55.038552 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe47cb31-d165-4927-9ca8-a335544fb1b3 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:55.038552 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe47cb31-d165-4927-9ca8-a335544fb1b3 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:55.038552 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe47cb31-d165-4927-9ca8-a335544fb1b3 None admin] PATH_INFO: `/v3/roles` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:55.039280 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe47cb31-d165-4927-9ca8-a335544fb1b3 None admin] RBAC: Adding query filter params (name=member) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:42:55.039847 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe47cb31-d165-4927-9ca8-a335544fb1b3 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:42:55.041874 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe47cb31-d165-4927-9ca8-a335544fb1b3 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:42:55.053386 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-560dd4de-0835-488f-8fd1-286f055cb23a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:42:55.054154 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 176/351] 10.222.0.27 () {56 vars in 1101 bytes} [Tue Aug 30 01:42:54 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 220 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:42:55.054263 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-560dd4de-0835-488f-8fd1-286f055cb23a None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:55.054438 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-560dd4de-0835-488f-8fd1-286f055cb23a None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:55.055187 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-560dd4de-0835-488f-8fd1-286f055cb23a None admin] PATH_INFO: `/v3/users/7075d18eee8744e8a1a4fd95076896e1` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:55.055431 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe47cb31-d165-4927-9ca8-a335544fb1b3 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:42:55.062869 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-560dd4de-0835-488f-8fd1-286f055cb23a None admin] RBAC: Authorizing `identity:get_user(user_id=7075d18eee8744e8a1a4fd95076896e1)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:42:55.064511 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-560dd4de-0835-488f-8fd1-286f055cb23a None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:42:55.069909 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 176/352] 10.222.0.27 () {56 vars in 1177 bytes} [Tue Aug 30 01:42:54 2022] GET /identity/v3/users/7075d18eee8744e8a1a4fd95076896e1 => generated 293 bytes in 203 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:42:55.070329 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-560dd4de-0835-488f-8fd1-286f055cb23a None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:42:55.145181 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:42:55.191427 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe47cb31-d165-4927-9ca8-a335544fb1b3 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:42:55.199366 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-560dd4de-0835-488f-8fd1-286f055cb23a None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:42:55.236056 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-317c77f0-f2c0-4f52-9853-ea7f34eab947 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:42:55.236919 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-317c77f0-f2c0-4f52-9853-ea7f34eab947 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:55.237261 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-317c77f0-f2c0-4f52-9853-ea7f34eab947 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:55.237565 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-317c77f0-f2c0-4f52-9853-ea7f34eab947 None admin] PATH_INFO: `/v3/roles` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:55.238234 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-317c77f0-f2c0-4f52-9853-ea7f34eab947 None admin] RBAC: Adding query filter params (name=member) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:42:55.238539 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-317c77f0-f2c0-4f52-9853-ea7f34eab947 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:42:55.239913 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-317c77f0-f2c0-4f52-9853-ea7f34eab947 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:42:55.240705 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-025d0705-a26b-4d81-8630-ae834b24e2f3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:42:55.241602 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-025d0705-a26b-4d81-8630-ae834b24e2f3 None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:55.242137 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-025d0705-a26b-4d81-8630-ae834b24e2f3 None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:55.242610 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-025d0705-a26b-4d81-8630-ae834b24e2f3 None admin] PATH_INFO: `/v3/projects/90818f998d5f4520971eacf6ccc830ce` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:55.243966 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-025d0705-a26b-4d81-8630-ae834b24e2f3 None admin] RBAC: Authorizing `identity:get_project(project_id=90818f998d5f4520971eacf6ccc830ce)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:42:55.244177 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:42:55.246099 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-025d0705-a26b-4d81-8630-ae834b24e2f3 None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:42:55.255150 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 177/353] 10.222.0.27 () {56 vars in 1186 bytes} [Tue Aug 30 01:42:55 2022] GET /identity/v3/projects/90818f998d5f4520971eacf6ccc830ce => generated 298 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:42:55.255313 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 177/354] 10.222.0.27 () {56 vars in 1101 bytes} [Tue Aug 30 01:42:55 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 199 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:42:55.255391 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5fd7ad8-a074-4c74-a3d5-813fdec70322 None None] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:55.255391 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5fd7ad8-a074-4c74-a3d5-813fdec70322 None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:55.255391 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5fd7ad8-a074-4c74-a3d5-813fdec70322 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:55.261206 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-317c77f0-f2c0-4f52-9853-ea7f34eab947 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:42:55.279084 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.auth.core [None req-e5fd7ad8-a074-4c74-a3d5-813fdec70322 None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81578) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 01:42:55.280609 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-e5fd7ad8-a074-4c74-a3d5-813fdec70322 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:42:55.372447 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 178/355] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Aug 30 01:42:55 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 120 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 30 01:42:55.379374 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5fd7ad8-a074-4c74-a3d5-813fdec70322 None None] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:42:55.403710 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-317c77f0-f2c0-4f52-9853-ea7f34eab947 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:42:55.447494 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-346816a6-bd76-4c6c-b1b0-0d26a5cc126a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:42:55.448111 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-346816a6-bd76-4c6c-b1b0-0d26a5cc126a None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:55.448391 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-346816a6-bd76-4c6c-b1b0-0d26a5cc126a None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:55.448673 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-346816a6-bd76-4c6c-b1b0-0d26a5cc126a None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:55.469777 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:42:55.530250 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:42:55.534034 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-346816a6-bd76-4c6c-b1b0-0d26a5cc126a None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:42:55.534647 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5fd7ad8-a074-4c74-a3d5-813fdec70322 None None] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:42:55.535559 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-346816a6-bd76-4c6c-b1b0-0d26a5cc126a None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:42:55.577122 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f70aab4-5d7c-485f-b7ed-f35b01afd687 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:42:55.577772 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f70aab4-5d7c-485f-b7ed-f35b01afd687 None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:55.578053 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f70aab4-5d7c-485f-b7ed-f35b01afd687 None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:55.578381 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f70aab4-5d7c-485f-b7ed-f35b01afd687 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:55.579103 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f70aab4-5d7c-485f-b7ed-f35b01afd687 None admin] RBAC: Adding query filter params (role.id=ffde3a55b5dd43149c18c72b7f2fe6ae, user.id=7075d18eee8744e8a1a4fd95076896e1, scope.system=all) {{(pid=81578) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:42:55.579464 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f70aab4-5d7c-485f-b7ed-f35b01afd687 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:42:55.581330 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f70aab4-5d7c-485f-b7ed-f35b01afd687 None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:42:55.600805 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 179/356] 10.222.0.27 () {56 vars in 1308 bytes} [Tue Aug 30 01:42:55 2022] GET /identity/v3/role_assignments?user.id=7075d18eee8744e8a1a4fd95076896e1&scope.system=all&role.id=ffde3a55b5dd43149c18c72b7f2fe6ae => generated 510 bytes in 228 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:42:55.605571 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f70aab4-5d7c-485f-b7ed-f35b01afd687 None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:42:55.663280 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 178/357] 10.222.0.27 () {58 vars in 1282 bytes} [Tue Aug 30 01:42:55 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 410 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 30 01:42:55.664693 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-346816a6-bd76-4c6c-b1b0-0d26a5cc126a None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:42:55.778159 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f70aab4-5d7c-485f-b7ed-f35b01afd687 None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:42:55.793917 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:42:55.803162 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:42:55.807969 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-346816a6-bd76-4c6c-b1b0-0d26a5cc126a None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:42:55.821722 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d79524e4-1804-4dce-a71c-400ee5ca727d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:42:55.822439 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d79524e4-1804-4dce-a71c-400ee5ca727d None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:55.822732 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d79524e4-1804-4dce-a71c-400ee5ca727d None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:55.822992 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d79524e4-1804-4dce-a71c-400ee5ca727d None admin] PATH_INFO: `/v3/groups/40df2032c69847bbaab1d9363f127d52` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:55.829304 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d79524e4-1804-4dce-a71c-400ee5ca727d None admin] RBAC: Authorizing `identity:get_group(group_id=40df2032c69847bbaab1d9363f127d52)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:42:55.830905 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d79524e4-1804-4dce-a71c-400ee5ca727d None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:42:55.833508 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 180/358] 10.222.0.27 () {56 vars in 1180 bytes} [Tue Aug 30 01:42:55 2022] GET /identity/v3/groups/40df2032c69847bbaab1d9363f127d52 => generated 226 bytes in 231 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:42:55.834687 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d68ac173-bfde-4450-8514-fa618a4f0347 None None] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:55.834964 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d68ac173-bfde-4450-8514-fa618a4f0347 None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:55.835234 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d68ac173-bfde-4450-8514-fa618a4f0347 None None] PATH_INFO: `/` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:55.837268 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 181/359] 10.222.0.27 () {54 vars in 927 bytes} [Tue Aug 30 01:42:55 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 01:42:55.838004 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d68ac173-bfde-4450-8514-fa618a4f0347 None None] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:42:55.859242 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5556ef9f-ff8e-4bfb-bfb1-588ab0cb1411 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:42:55.859866 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5556ef9f-ff8e-4bfb-bfb1-588ab0cb1411 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:55.860143 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5556ef9f-ff8e-4bfb-bfb1-588ab0cb1411 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:55.860424 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5556ef9f-ff8e-4bfb-bfb1-588ab0cb1411 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:55.940468 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5556ef9f-ff8e-4bfb-bfb1-588ab0cb1411 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:42:55.942789 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5556ef9f-ff8e-4bfb-bfb1-588ab0cb1411 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:42:55.957913 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d68ac173-bfde-4450-8514-fa618a4f0347 None None] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:42:56.000591 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92b9a952-7de0-4496-b0c8-8f1eba4ff7ae None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:42:56.001222 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92b9a952-7de0-4496-b0c8-8f1eba4ff7ae None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:56.001528 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92b9a952-7de0-4496-b0c8-8f1eba4ff7ae None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:56.001803 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92b9a952-7de0-4496-b0c8-8f1eba4ff7ae None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:56.013978 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92b9a952-7de0-4496-b0c8-8f1eba4ff7ae None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 30 01:42:56.019224 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application [None req-92b9a952-7de0-4496-b0c8-8f1eba4ff7ae None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Aug 30 01:42:56.019224 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 01:42:56.019224 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 01:42:56.019224 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 01:42:56.019224 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 01:42:56.019224 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 01:42:56.019224 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 01:42:56.019224 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 01:42:56.019224 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 01:42:56.019224 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 01:42:56.019224 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 01:42:56.019224 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 01:42:56.019224 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 30 01:42:56.019224 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 30 01:42:56.019224 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 30 01:42:56.019224 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 30 01:42:56.020283 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 30 01:42:56.020283 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 30 01:42:56.020283 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 30 01:42:56.020283 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 30 01:42:56.020283 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 01:42:56.020283 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 01:42:56.020283 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 01:42:56.020283 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 01:42:56.020283 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 01:42:56.020283 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 01:42:56.020283 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 01:42:56.020283 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application with Lock( Aug 30 01:42:56.020283 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 01:42:56.020283 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._enter() Aug 30 01:42:56.020283 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 01:42:56.020283 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 01:42:56.021433 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 01:42:56.021433 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.creator() Aug 30 01:42:56.021433 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 01:42:56.021433 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application created_value = creator( Aug 30 01:42:56.021433 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 30 01:42:56.021433 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 30 01:42:56.021433 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 30 01:42:56.021433 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 30 01:42:56.021433 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 30 01:42:56.021433 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 30 01:42:56.021433 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Aug 30 01:42:56.021433 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application Aug 30 01:42:56.021433 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 182/360] 10.222.0.27 () {56 vars in 1114 bytes} [Tue Aug 30 01:42:55 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 183 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 01:42:56.021433 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92b9a952-7de0-4496-b0c8-8f1eba4ff7ae None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:42:56.052060 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:42:56.057583 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 179/361] 10.222.0.27 () {58 vars in 1282 bytes} [Tue Aug 30 01:42:55 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 394 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 30 01:42:56.059001 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b83bdc64-128a-432a-8b07-344217618d5a None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:56.059292 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b83bdc64-128a-432a-8b07-344217618d5a None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:56.059581 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b83bdc64-128a-432a-8b07-344217618d5a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:56.078931 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-b83bdc64-128a-432a-8b07-344217618d5a None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 01:42:56.080139 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-b83bdc64-128a-432a-8b07-344217618d5a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:42:56.084941 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:42:56.156515 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92b9a952-7de0-4496-b0c8-8f1eba4ff7ae None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:42:56.157838 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 180/362] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Aug 30 01:42:56 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 30 01:42:56.158618 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b83bdc64-128a-432a-8b07-344217618d5a None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:42:56.204082 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c4797ac-5769-4c9c-aaca-ad062e6622a3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:42:56.204722 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c4797ac-5769-4c9c-aaca-ad062e6622a3 None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:56.205005 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c4797ac-5769-4c9c-aaca-ad062e6622a3 None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:56.205278 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c4797ac-5769-4c9c-aaca-ad062e6622a3 None admin] PATH_INFO: `/v3/projects/90818f998d5f4520971eacf6ccc830ce` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:56.206454 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c4797ac-5769-4c9c-aaca-ad062e6622a3 None admin] RBAC: Authorizing `identity:get_project(project_id=90818f998d5f4520971eacf6ccc830ce)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:42:56.207900 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c4797ac-5769-4c9c-aaca-ad062e6622a3 None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:42:56.209678 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 183/363] 10.222.0.27 () {56 vars in 1186 bytes} [Tue Aug 30 01:42:56 2022] GET /identity/v3/projects/90818f998d5f4520971eacf6ccc830ce => generated 298 bytes in 189 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:42:56.210437 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c4797ac-5769-4c9c-aaca-ad062e6622a3 None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:42:56.293150 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b83bdc64-128a-432a-8b07-344217618d5a None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:42:56.332849 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:42:56.335261 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c4797ac-5769-4c9c-aaca-ad062e6622a3 None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:42:56.342123 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7488330a-4b05-413b-a315-f3c947063cb2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:42:56.342739 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7488330a-4b05-413b-a315-f3c947063cb2 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:56.343019 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7488330a-4b05-413b-a315-f3c947063cb2 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:56.343292 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7488330a-4b05-413b-a315-f3c947063cb2 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:56.346332 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7488330a-4b05-413b-a315-f3c947063cb2 None admin] RBAC: Adding query filter params (group.id=40df2032c69847bbaab1d9363f127d52, role.id=4621068972ba4f79bad104f956db04aa, scope.project.id=90818f998d5f4520971eacf6ccc830ce) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:42:56.346332 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7488330a-4b05-413b-a315-f3c947063cb2 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:42:56.346508 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:42:56.346594 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7488330a-4b05-413b-a315-f3c947063cb2 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:42:56.353598 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 181/364] 10.222.0.27 () {56 vars in 1376 bytes} [Tue Aug 30 01:42:56 2022] GET /identity/v3/role_assignments?group.id=40df2032c69847bbaab1d9363f127d52&scope.project.id=90818f998d5f4520971eacf6ccc830ce&role.id=4621068972ba4f79bad104f956db04aa => generated 260 bytes in 195 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:42:56.354273 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7488330a-4b05-413b-a315-f3c947063cb2 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:42:56.381103 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f21fe0a4-03b1-418c-b2df-7e187e581382 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:42:56.381811 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f21fe0a4-03b1-418c-b2df-7e187e581382 None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:56.382020 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f21fe0a4-03b1-418c-b2df-7e187e581382 None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:56.382408 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f21fe0a4-03b1-418c-b2df-7e187e581382 None admin] PATH_INFO: `/v3/roles` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:56.384270 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f21fe0a4-03b1-418c-b2df-7e187e581382 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81578) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:42:56.384270 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f21fe0a4-03b1-418c-b2df-7e187e581382 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:42:56.386408 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f21fe0a4-03b1-418c-b2df-7e187e581382 None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:42:56.396604 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 184/365] 10.222.0.27 () {56 vars in 1111 bytes} [Tue Aug 30 01:42:56 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 186 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:42:56.397280 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f21fe0a4-03b1-418c-b2df-7e187e581382 None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:42:56.506526 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7488330a-4b05-413b-a315-f3c947063cb2 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:42:56.538620 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f21fe0a4-03b1-418c-b2df-7e187e581382 None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:42:56.550002 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e16c5a7-5e1a-43d9-860f-6e906842bb42 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:42:56.550689 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e16c5a7-5e1a-43d9-860f-6e906842bb42 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:56.550962 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e16c5a7-5e1a-43d9-860f-6e906842bb42 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:56.551227 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e16c5a7-5e1a-43d9-860f-6e906842bb42 None admin] PATH_INFO: `/v3/projects/67913d5682a443069a5eed96f179751f` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:56.552392 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e16c5a7-5e1a-43d9-860f-6e906842bb42 None admin] RBAC: Authorizing `identity:get_project(project_id=67913d5682a443069a5eed96f179751f)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:42:56.553929 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e16c5a7-5e1a-43d9-860f-6e906842bb42 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:42:56.556773 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 182/366] 10.222.0.27 () {56 vars in 1186 bytes} [Tue Aug 30 01:42:56 2022] GET /identity/v3/projects/67913d5682a443069a5eed96f179751f => generated 302 bytes in 202 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:42:56.557582 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e16c5a7-5e1a-43d9-860f-6e906842bb42 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:42:56.585404 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e8ab0e0-c5ad-458c-8d44-061d4b4a079c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:42:56.586017 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e8ab0e0-c5ad-458c-8d44-061d4b4a079c None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:56.586297 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e8ab0e0-c5ad-458c-8d44-061d4b4a079c None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:56.586561 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e8ab0e0-c5ad-458c-8d44-061d4b4a079c None admin] PATH_INFO: `/v3/groups/40df2032c69847bbaab1d9363f127d52` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:56.587851 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e8ab0e0-c5ad-458c-8d44-061d4b4a079c None admin] RBAC: Authorizing `identity:get_group(group_id=40df2032c69847bbaab1d9363f127d52)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:42:56.589290 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e8ab0e0-c5ad-458c-8d44-061d4b4a079c None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:42:56.590900 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 185/367] 10.222.0.27 () {56 vars in 1180 bytes} [Tue Aug 30 01:42:56 2022] GET /identity/v3/groups/40df2032c69847bbaab1d9363f127d52 => generated 226 bytes in 194 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:42:56.591674 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e8ab0e0-c5ad-458c-8d44-061d4b4a079c None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:42:56.629354 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:42:56.638084 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:42:56.685061 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e16c5a7-5e1a-43d9-860f-6e906842bb42 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:42:56.723896 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e8ab0e0-c5ad-458c-8d44-061d4b4a079c None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:42:56.737630 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd468b5a-c6ad-4391-948b-ab2b9ff2178b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:42:56.738522 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd468b5a-c6ad-4391-948b-ab2b9ff2178b None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:56.738802 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd468b5a-c6ad-4391-948b-ab2b9ff2178b None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:56.739070 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd468b5a-c6ad-4391-948b-ab2b9ff2178b None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:56.764391 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15453550-ae98-464d-9822-15d10a2c2773 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:42:56.765365 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15453550-ae98-464d-9822-15d10a2c2773 None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:56.765894 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15453550-ae98-464d-9822-15d10a2c2773 None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:56.766173 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15453550-ae98-464d-9822-15d10a2c2773 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:56.778879 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15453550-ae98-464d-9822-15d10a2c2773 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 30 01:42:56.786361 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application [None req-15453550-ae98-464d-9822-15d10a2c2773 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 30 01:42:56.786361 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 01:42:56.786361 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 01:42:56.786361 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 01:42:56.786361 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 01:42:56.786361 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 01:42:56.786361 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 01:42:56.786361 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 01:42:56.786361 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 01:42:56.786361 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 01:42:56.786361 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 01:42:56.786361 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 01:42:56.786361 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 30 01:42:56.786361 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 30 01:42:56.786361 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 30 01:42:56.786361 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 30 01:42:56.787334 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 30 01:42:56.787334 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 30 01:42:56.787334 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 30 01:42:56.787334 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 30 01:42:56.787334 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 01:42:56.787334 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 01:42:56.787334 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 01:42:56.787334 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 01:42:56.787334 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 01:42:56.787334 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 01:42:56.787334 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 01:42:56.787334 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application with Lock( Aug 30 01:42:56.787334 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 01:42:56.787334 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._enter() Aug 30 01:42:56.787334 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 01:42:56.787334 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 01:42:56.788157 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 01:42:56.788157 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.creator() Aug 30 01:42:56.788157 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 01:42:56.788157 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application created_value = creator( Aug 30 01:42:56.788157 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 30 01:42:56.788157 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 30 01:42:56.788157 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 30 01:42:56.788157 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 30 01:42:56.788157 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 30 01:42:56.788157 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 30 01:42:56.788157 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 30 01:42:56.788157 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application Aug 30 01:42:56.788157 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 186/368] 10.222.0.27 () {56 vars in 1096 bytes} [Tue Aug 30 01:42:56 2022] GET /identity/v3/roles/admin => generated 83 bytes in 196 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 01:42:56.790160 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15453550-ae98-464d-9822-15d10a2c2773 None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:42:56.842914 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd468b5a-c6ad-4391-948b-ab2b9ff2178b None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:42:56.844314 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd468b5a-c6ad-4391-948b-ab2b9ff2178b None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:42:56.917131 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:42:56.924817 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15453550-ae98-464d-9822-15d10a2c2773 None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:42:56.928668 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:42:56.969392 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8b16260-c517-4514-9c45-f412526cdac4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:42:56.970120 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8b16260-c517-4514-9c45-f412526cdac4 None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:56.970525 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8b16260-c517-4514-9c45-f412526cdac4 None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:56.970648 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8b16260-c517-4514-9c45-f412526cdac4 None admin] PATH_INFO: `/v3/groups/40df2032c69847bbaab1d9363f127d52` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:56.971765 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8b16260-c517-4514-9c45-f412526cdac4 None admin] RBAC: Authorizing `identity:get_group(group_id=40df2032c69847bbaab1d9363f127d52)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:42:56.973188 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8b16260-c517-4514-9c45-f412526cdac4 None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:42:56.974940 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 187/369] 10.222.0.27 () {56 vars in 1180 bytes} [Tue Aug 30 01:42:56 2022] GET /identity/v3/groups/40df2032c69847bbaab1d9363f127d52 => generated 226 bytes in 186 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:42:56.978135 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8b16260-c517-4514-9c45-f412526cdac4 None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:42:56.985500 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 183/370] 10.222.0.27 () {58 vars in 1282 bytes} [Tue Aug 30 01:42:56 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 428 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 30 01:42:56.986451 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd468b5a-c6ad-4391-948b-ab2b9ff2178b None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:42:57.100107 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8b16260-c517-4514-9c45-f412526cdac4 None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:42:57.111503 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd468b5a-c6ad-4391-948b-ab2b9ff2178b None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:42:57.138738 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-659fc60f-ccce-4492-a336-ed4445822518 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:42:57.139427 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-659fc60f-ccce-4492-a336-ed4445822518 None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:57.139770 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-659fc60f-ccce-4492-a336-ed4445822518 None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:57.140106 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-659fc60f-ccce-4492-a336-ed4445822518 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:57.140826 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-659fc60f-ccce-4492-a336-ed4445822518 None admin] RBAC: Adding query filter params (group.id=40df2032c69847bbaab1d9363f127d52, role.id=1fb81afcb9f245f3b989add94598a063, scope.project.id=90818f998d5f4520971eacf6ccc830ce) {{(pid=81578) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:42:57.141127 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-659fc60f-ccce-4492-a336-ed4445822518 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:42:57.142575 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-659fc60f-ccce-4492-a336-ed4445822518 None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:42:57.147786 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 188/371] 10.222.0.27 () {56 vars in 1376 bytes} [Tue Aug 30 01:42:56 2022] GET /identity/v3/role_assignments?group.id=40df2032c69847bbaab1d9363f127d52&scope.project.id=90818f998d5f4520971eacf6ccc830ce&role.id=1fb81afcb9f245f3b989add94598a063 => generated 260 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:42:57.148310 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0536bd31-2865-47b5-ab4d-c75b545e18bd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:42:57.148704 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-659fc60f-ccce-4492-a336-ed4445822518 None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:42:57.149374 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0536bd31-2865-47b5-ab4d-c75b545e18bd None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:57.149771 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0536bd31-2865-47b5-ab4d-c75b545e18bd None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:57.150138 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0536bd31-2865-47b5-ab4d-c75b545e18bd None admin] PATH_INFO: `/v3/roles` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:57.150911 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0536bd31-2865-47b5-ab4d-c75b545e18bd None admin] RBAC: Adding query filter params (name=admin) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:42:57.151316 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0536bd31-2865-47b5-ab4d-c75b545e18bd None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:42:57.152842 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0536bd31-2865-47b5-ab4d-c75b545e18bd None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:42:57.165643 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 184/372] 10.222.0.27 () {56 vars in 1099 bytes} [Tue Aug 30 01:42:56 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 180 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:42:57.166786 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0536bd31-2865-47b5-ab4d-c75b545e18bd None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:42:57.196870 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:42:57.199269 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:42:57.287759 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-659fc60f-ccce-4492-a336-ed4445822518 None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:42:57.296751 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0536bd31-2865-47b5-ab4d-c75b545e18bd None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:42:57.330667 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ff18529-bfdb-4a61-a889-0fdc8f4671bc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:42:57.331749 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ff18529-bfdb-4a61-a889-0fdc8f4671bc None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:57.332131 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ff18529-bfdb-4a61-a889-0fdc8f4671bc None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:57.332480 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ff18529-bfdb-4a61-a889-0fdc8f4671bc None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:57.333282 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ff18529-bfdb-4a61-a889-0fdc8f4671bc None admin] RBAC: Adding query filter params (group.id=40df2032c69847bbaab1d9363f127d52, role.id=1fb81afcb9f245f3b989add94598a063, scope.project.id=67913d5682a443069a5eed96f179751f) {{(pid=81578) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:42:57.333692 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ff18529-bfdb-4a61-a889-0fdc8f4671bc None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:42:57.335213 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ff18529-bfdb-4a61-a889-0fdc8f4671bc None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:42:57.349485 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 189/373] 10.222.0.27 () {56 vars in 1376 bytes} [Tue Aug 30 01:42:57 2022] GET /identity/v3/role_assignments?group.id=40df2032c69847bbaab1d9363f127d52&scope.project.id=67913d5682a443069a5eed96f179751f&role.id=1fb81afcb9f245f3b989add94598a063 => generated 260 bytes in 194 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:42:57.349485 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ff18529-bfdb-4a61-a889-0fdc8f4671bc None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:42:57.354860 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff6ebcfa-536c-484d-b059-f652cdea3f67 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:42:57.359304 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff6ebcfa-536c-484d-b059-f652cdea3f67 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:57.359797 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff6ebcfa-536c-484d-b059-f652cdea3f67 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:57.360179 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff6ebcfa-536c-484d-b059-f652cdea3f67 None admin] PATH_INFO: `/v3/projects/67913d5682a443069a5eed96f179751f` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:57.363354 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff6ebcfa-536c-484d-b059-f652cdea3f67 None admin] RBAC: Authorizing `identity:get_project(project_id=67913d5682a443069a5eed96f179751f)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:42:57.365028 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff6ebcfa-536c-484d-b059-f652cdea3f67 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:42:57.366967 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 185/374] 10.222.0.27 () {56 vars in 1186 bytes} [Tue Aug 30 01:42:57 2022] GET /identity/v3/projects/67913d5682a443069a5eed96f179751f => generated 302 bytes in 201 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:42:57.373445 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff6ebcfa-536c-484d-b059-f652cdea3f67 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:42:57.483168 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:42:57.494380 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff6ebcfa-536c-484d-b059-f652cdea3f67 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:42:57.532128 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf67ea2e-ebb0-4138-b488-bdb6285e9b64 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:42:57.532744 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf67ea2e-ebb0-4138-b488-bdb6285e9b64 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:57.533064 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf67ea2e-ebb0-4138-b488-bdb6285e9b64 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:57.533336 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf67ea2e-ebb0-4138-b488-bdb6285e9b64 None admin] PATH_INFO: `/v3/groups/40df2032c69847bbaab1d9363f127d52` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:57.534493 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf67ea2e-ebb0-4138-b488-bdb6285e9b64 None admin] RBAC: Authorizing `identity:get_group(group_id=40df2032c69847bbaab1d9363f127d52)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:42:57.535961 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf67ea2e-ebb0-4138-b488-bdb6285e9b64 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:42:57.542967 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 186/375] 10.222.0.27 () {56 vars in 1180 bytes} [Tue Aug 30 01:42:57 2022] GET /identity/v3/groups/40df2032c69847bbaab1d9363f127d52 => generated 226 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:42:57.544241 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf67ea2e-ebb0-4138-b488-bdb6285e9b64 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:42:57.567912 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:42:57.582449 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ff18529-bfdb-4a61-a889-0fdc8f4671bc None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:42:57.632183 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4920f91-b982-4423-96f8-6de69bddd8a7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:42:57.635304 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4920f91-b982-4423-96f8-6de69bddd8a7 None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:57.635593 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4920f91-b982-4423-96f8-6de69bddd8a7 None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:57.635866 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4920f91-b982-4423-96f8-6de69bddd8a7 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:57.698090 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf67ea2e-ebb0-4138-b488-bdb6285e9b64 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:42:57.738810 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4920f91-b982-4423-96f8-6de69bddd8a7 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:42:57.740267 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4920f91-b982-4423-96f8-6de69bddd8a7 None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:42:57.773767 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-759de258-5504-4158-82fa-af8ed7a48bfb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:42:57.774550 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-759de258-5504-4158-82fa-af8ed7a48bfb None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:57.774829 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-759de258-5504-4158-82fa-af8ed7a48bfb None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:57.775096 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-759de258-5504-4158-82fa-af8ed7a48bfb None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:57.775832 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-759de258-5504-4158-82fa-af8ed7a48bfb None admin] RBAC: Adding query filter params (group.id=40df2032c69847bbaab1d9363f127d52, role.id=4621068972ba4f79bad104f956db04aa, scope.project.id=67913d5682a443069a5eed96f179751f) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:42:57.776142 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-759de258-5504-4158-82fa-af8ed7a48bfb None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:42:57.777617 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-759de258-5504-4158-82fa-af8ed7a48bfb None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:42:57.790859 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 187/376] 10.222.0.27 () {56 vars in 1376 bytes} [Tue Aug 30 01:42:57 2022] GET /identity/v3/role_assignments?group.id=40df2032c69847bbaab1d9363f127d52&scope.project.id=67913d5682a443069a5eed96f179751f&role.id=4621068972ba4f79bad104f956db04aa => generated 260 bytes in 247 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:42:57.870790 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:42:57.881247 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 190/377] 10.222.0.27 () {58 vars in 1282 bytes} [Tue Aug 30 01:42:57 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 538 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 30 01:42:57.887664 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-759de258-5504-4158-82fa-af8ed7a48bfb None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:42:57.940084 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:42:58.045323 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-759de258-5504-4158-82fa-af8ed7a48bfb None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:42:58.093031 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff4e9c26-07b4-4ce3-8534-62037299bb5d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:42:58.094112 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff4e9c26-07b4-4ce3-8534-62037299bb5d None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:58.094586 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff4e9c26-07b4-4ce3-8534-62037299bb5d None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:58.095075 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff4e9c26-07b4-4ce3-8534-62037299bb5d None admin] PATH_INFO: `/v3/projects/70bc9f42895844308fb4b9192314f80b` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:58.096403 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff4e9c26-07b4-4ce3-8534-62037299bb5d None admin] RBAC: Authorizing `identity:get_project(project_id=70bc9f42895844308fb4b9192314f80b)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:42:58.098064 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff4e9c26-07b4-4ce3-8534-62037299bb5d None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:42:58.099888 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 188/378] 10.222.0.27 () {56 vars in 1186 bytes} [Tue Aug 30 01:42:57 2022] GET /identity/v3/projects/70bc9f42895844308fb4b9192314f80b => generated 344 bytes in 213 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:42:58.106523 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4920f91-b982-4423-96f8-6de69bddd8a7 None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:42:58.277716 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4920f91-b982-4423-96f8-6de69bddd8a7 None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:42:58.328970 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcc74485-a56c-4c84-a3d3-99d56bf81aac None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:42:58.329641 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcc74485-a56c-4c84-a3d3-99d56bf81aac None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:58.329926 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcc74485-a56c-4c84-a3d3-99d56bf81aac None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:58.330199 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcc74485-a56c-4c84-a3d3-99d56bf81aac None admin] PATH_INFO: `/v3/groups/e130c65bd71a4dc983e47fda13b8d556` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:58.340071 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bcc74485-a56c-4c84-a3d3-99d56bf81aac None admin] RBAC: Authorizing `identity:get_group(group_id=e130c65bd71a4dc983e47fda13b8d556)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:42:58.341540 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bcc74485-a56c-4c84-a3d3-99d56bf81aac None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:42:58.343194 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 191/379] 10.222.0.27 () {56 vars in 1180 bytes} [Tue Aug 30 01:42:58 2022] GET /identity/v3/groups/e130c65bd71a4dc983e47fda13b8d556 => generated 229 bytes in 237 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:42:58.349161 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff4e9c26-07b4-4ce3-8534-62037299bb5d None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:42:58.512188 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:42:58.522713 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff4e9c26-07b4-4ce3-8534-62037299bb5d None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:42:58.563821 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4053c884-c31f-4865-84b6-e8d488870044 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:42:58.564465 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4053c884-c31f-4865-84b6-e8d488870044 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:58.564691 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4053c884-c31f-4865-84b6-e8d488870044 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:58.564968 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4053c884-c31f-4865-84b6-e8d488870044 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:58.565729 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4053c884-c31f-4865-84b6-e8d488870044 None admin] RBAC: Adding query filter params (group.id=e130c65bd71a4dc983e47fda13b8d556, role.id=ead5fac86a0747ea8823a82efcea8ed7, scope.project.id=70bc9f42895844308fb4b9192314f80b) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:42:58.566040 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4053c884-c31f-4865-84b6-e8d488870044 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:42:58.567473 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4053c884-c31f-4865-84b6-e8d488870044 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:42:58.573988 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 189/380] 10.222.0.27 () {56 vars in 1376 bytes} [Tue Aug 30 01:42:58 2022] GET /identity/v3/role_assignments?group.id=e130c65bd71a4dc983e47fda13b8d556&scope.project.id=70bc9f42895844308fb4b9192314f80b&role.id=ead5fac86a0747ea8823a82efcea8ed7 => generated 260 bytes in 225 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:42:58.599321 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-677655bc-b37b-4ef9-a178-095d7df5a718 None None] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:58.599872 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-677655bc-b37b-4ef9-a178-095d7df5a718 None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:58.600359 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-677655bc-b37b-4ef9-a178-095d7df5a718 None None] PATH_INFO: `/` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:58.603988 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 192/381] 10.222.0.27 () {54 vars in 927 bytes} [Tue Aug 30 01:42:58 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 01:42:58.608664 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac36ed57-4b8a-46f9-a20f-ddff37892a7a None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:58.608953 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac36ed57-4b8a-46f9-a20f-ddff37892a7a None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:58.609229 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac36ed57-4b8a-46f9-a20f-ddff37892a7a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:58.633064 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-ac36ed57-4b8a-46f9-a20f-ddff37892a7a None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 01:42:58.634333 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-ac36ed57-4b8a-46f9-a20f-ddff37892a7a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:42:58.749556 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 190/382] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Aug 30 01:42:58 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 142 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 30 01:42:59.114229 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-677655bc-b37b-4ef9-a178-095d7df5a718 None None] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:42:59.183946 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:42:59.246163 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-677655bc-b37b-4ef9-a178-095d7df5a718 None None] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:42:59.292327 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a5add98-f7d0-455f-8275-57efb1c8cf99 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:42:59.292964 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a5add98-f7d0-455f-8275-57efb1c8cf99 None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:59.293261 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a5add98-f7d0-455f-8275-57efb1c8cf99 None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:59.293539 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a5add98-f7d0-455f-8275-57efb1c8cf99 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:59.307096 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a5add98-f7d0-455f-8275-57efb1c8cf99 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 30 01:42:59.312059 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application [None req-5a5add98-f7d0-455f-8275-57efb1c8cf99 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Aug 30 01:42:59.312059 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 01:42:59.312059 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 01:42:59.312059 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 01:42:59.312059 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 01:42:59.312059 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 01:42:59.312059 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 01:42:59.312059 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 01:42:59.312059 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 01:42:59.312059 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 01:42:59.312059 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 01:42:59.312059 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 01:42:59.312059 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 30 01:42:59.312059 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 30 01:42:59.312059 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 30 01:42:59.312059 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 30 01:42:59.312963 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 30 01:42:59.312963 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 30 01:42:59.312963 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 30 01:42:59.312963 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 30 01:42:59.312963 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 01:42:59.312963 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 01:42:59.312963 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 01:42:59.312963 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 01:42:59.312963 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 01:42:59.312963 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 01:42:59.312963 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 01:42:59.312963 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application with Lock( Aug 30 01:42:59.312963 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 01:42:59.312963 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._enter() Aug 30 01:42:59.312963 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 01:42:59.312963 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 01:42:59.313800 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 01:42:59.313800 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.creator() Aug 30 01:42:59.313800 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 01:42:59.313800 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application created_value = creator( Aug 30 01:42:59.313800 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 30 01:42:59.313800 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 30 01:42:59.313800 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 30 01:42:59.313800 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 30 01:42:59.313800 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 30 01:42:59.313800 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 30 01:42:59.313800 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Aug 30 01:42:59.313800 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application Aug 30 01:42:59.313800 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 193/383] 10.222.0.27 () {56 vars in 1114 bytes} [Tue Aug 30 01:42:59 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 200 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 01:42:59.319091 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac36ed57-4b8a-46f9-a20f-ddff37892a7a None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:42:59.349489 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac6fc9f5-52b7-4f93-b09a-e3cd6e52abea None None] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:59.349787 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac6fc9f5-52b7-4f93-b09a-e3cd6e52abea None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:59.350048 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac6fc9f5-52b7-4f93-b09a-e3cd6e52abea None None] PATH_INFO: `/` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:59.352426 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 194/384] 10.222.0.27 () {54 vars in 927 bytes} [Tue Aug 30 01:42:59 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 30 01:42:59.357412 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-277b33f7-db1e-4232-8e39-40b25dc5005b None None] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:59.357725 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-277b33f7-db1e-4232-8e39-40b25dc5005b None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:59.357997 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-277b33f7-db1e-4232-8e39-40b25dc5005b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:59.392169 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.auth.core [None req-277b33f7-db1e-4232-8e39-40b25dc5005b None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81578) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 01:42:59.393325 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-277b33f7-db1e-4232-8e39-40b25dc5005b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:42:59.479203 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:42:59.515456 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac36ed57-4b8a-46f9-a20f-ddff37892a7a None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:42:59.517596 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 195/385] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Aug 30 01:42:59 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 155 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 30 01:42:59.574217 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-038a5206-6bcc-42ec-afff-bcd4ea9a7c13 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:42:59.574864 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-038a5206-6bcc-42ec-afff-bcd4ea9a7c13 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:59.575110 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-038a5206-6bcc-42ec-afff-bcd4ea9a7c13 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:59.575388 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-038a5206-6bcc-42ec-afff-bcd4ea9a7c13 None admin] PATH_INFO: `/v3/roles` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:59.576063 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-038a5206-6bcc-42ec-afff-bcd4ea9a7c13 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:42:59.576369 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-038a5206-6bcc-42ec-afff-bcd4ea9a7c13 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:42:59.577889 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-038a5206-6bcc-42ec-afff-bcd4ea9a7c13 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:42:59.592065 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 191/386] 10.222.0.27 () {56 vars in 1111 bytes} [Tue Aug 30 01:42:59 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 273 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:42:59.597856 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-277b33f7-db1e-4232-8e39-40b25dc5005b None None] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:42:59.693299 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:42:59.775562 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-277b33f7-db1e-4232-8e39-40b25dc5005b None None] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:42:59.830669 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e987ad9e-125d-4834-a66f-c02b33634fa9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:42:59.831325 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e987ad9e-125d-4834-a66f-c02b33634fa9 None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:42:59.831605 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-038a5206-6bcc-42ec-afff-bcd4ea9a7c13 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:42:59.831709 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e987ad9e-125d-4834-a66f-c02b33634fa9 None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:42:59.831874 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e987ad9e-125d-4834-a66f-c02b33634fa9 None admin] PATH_INFO: `/v3/groups/40df2032c69847bbaab1d9363f127d52` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:42:59.838213 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e987ad9e-125d-4834-a66f-c02b33634fa9 None admin] RBAC: Authorizing `identity:get_group(group_id=40df2032c69847bbaab1d9363f127d52)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:42:59.839692 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e987ad9e-125d-4834-a66f-c02b33634fa9 None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:42:59.841656 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 196/387] 10.222.0.27 () {56 vars in 1180 bytes} [Tue Aug 30 01:42:59 2022] GET /identity/v3/groups/40df2032c69847bbaab1d9363f127d52 => generated 226 bytes in 244 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:42:59.850254 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e987ad9e-125d-4834-a66f-c02b33634fa9 None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:00.011130 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-038a5206-6bcc-42ec-afff-bcd4ea9a7c13 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:00.026561 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e987ad9e-125d-4834-a66f-c02b33634fa9 None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:00.043425 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:43:00.055274 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aef1538c-5fb6-485d-9da6-0f6e63e46d4b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:00.056047 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aef1538c-5fb6-485d-9da6-0f6e63e46d4b None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:00.056423 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aef1538c-5fb6-485d-9da6-0f6e63e46d4b None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:00.056816 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aef1538c-5fb6-485d-9da6-0f6e63e46d4b None admin] PATH_INFO: `/v3/roles/member` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:00.067815 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aef1538c-5fb6-485d-9da6-0f6e63e46d4b None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 30 01:43:00.073698 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application [None req-aef1538c-5fb6-485d-9da6-0f6e63e46d4b None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Aug 30 01:43:00.073698 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 01:43:00.073698 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 01:43:00.073698 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 01:43:00.073698 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 01:43:00.073698 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 01:43:00.073698 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 01:43:00.073698 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 01:43:00.073698 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 01:43:00.073698 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 01:43:00.073698 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 01:43:00.073698 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 01:43:00.073698 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 30 01:43:00.073698 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 30 01:43:00.073698 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 30 01:43:00.073698 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 30 01:43:00.074789 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 30 01:43:00.074789 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 30 01:43:00.074789 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 30 01:43:00.074789 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 30 01:43:00.074789 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 01:43:00.074789 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 01:43:00.074789 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 01:43:00.074789 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 01:43:00.074789 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 01:43:00.074789 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 01:43:00.074789 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 01:43:00.074789 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application with Lock( Aug 30 01:43:00.074789 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 01:43:00.074789 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._enter() Aug 30 01:43:00.074789 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 01:43:00.074789 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 01:43:00.075855 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 01:43:00.075855 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.creator() Aug 30 01:43:00.075855 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 01:43:00.075855 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application created_value = creator( Aug 30 01:43:00.075855 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 30 01:43:00.075855 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 30 01:43:00.075855 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 30 01:43:00.075855 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 30 01:43:00.075855 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 30 01:43:00.075855 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 30 01:43:00.075855 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Aug 30 01:43:00.075855 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Aug 30 01:43:00.079297 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4dd9a7e7-9ba9-4b5a-b03d-470309306fc8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:00.079704 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 192/388] 10.222.0.27 () {56 vars in 1099 bytes} [Tue Aug 30 01:42:59 2022] GET /identity/v3/roles/member => generated 84 bytes in 244 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 30 01:43:00.079943 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dd9a7e7-9ba9-4b5a-b03d-470309306fc8 None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:00.080388 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dd9a7e7-9ba9-4b5a-b03d-470309306fc8 None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:00.080693 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dd9a7e7-9ba9-4b5a-b03d-470309306fc8 None admin] PATH_INFO: `/v3/projects/90818f998d5f4520971eacf6ccc830ce` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:00.081297 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96b6b086-d318-453a-9f7a-2d3e1b5215bd None None] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:00.081837 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4dd9a7e7-9ba9-4b5a-b03d-470309306fc8 None admin] RBAC: Authorizing `identity:get_project(project_id=90818f998d5f4520971eacf6ccc830ce)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:00.082053 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96b6b086-d318-453a-9f7a-2d3e1b5215bd None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:00.082280 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96b6b086-d318-453a-9f7a-2d3e1b5215bd None None] PATH_INFO: `/` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:00.083284 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4dd9a7e7-9ba9-4b5a-b03d-470309306fc8 None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:00.083762 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 193/389] 10.222.0.27 () {54 vars in 927 bytes} [Tue Aug 30 01:43:00 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 01:43:00.084832 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b03ce833-9854-4d55-936a-c84b90692538 None None] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:00.085167 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 197/390] 10.222.0.27 () {56 vars in 1186 bytes} [Tue Aug 30 01:42:59 2022] GET /identity/v3/projects/90818f998d5f4520971eacf6ccc830ce => generated 298 bytes in 235 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 30 01:43:00.089609 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4dd9a7e7-9ba9-4b5a-b03d-470309306fc8 None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:00.089765 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b03ce833-9854-4d55-936a-c84b90692538 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:00.089765 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b03ce833-9854-4d55-936a-c84b90692538 None None] PATH_INFO: `/` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:00.090816 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 194/391] 10.222.0.27 () {54 vars in 927 bytes} [Tue Aug 30 01:43:00 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 01:43:00.091994 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-246156a5-008a-4234-a9d5-c12e6087451a None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:00.092363 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-246156a5-008a-4234-a9d5-c12e6087451a None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:00.092641 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-246156a5-008a-4234-a9d5-c12e6087451a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:00.105962 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:43:00.111834 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-246156a5-008a-4234-a9d5-c12e6087451a None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 01:43:00.113034 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-246156a5-008a-4234-a9d5-c12e6087451a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:43:00.185808 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 195/392] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Aug 30 01:43:00 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 30 01:43:00.186684 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-246156a5-008a-4234-a9d5-c12e6087451a None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:00.214716 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4dd9a7e7-9ba9-4b5a-b03d-470309306fc8 None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:00.254349 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a33533fd-fdd4-4242-a987-529067802c3c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:00.254983 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a33533fd-fdd4-4242-a987-529067802c3c None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:00.255299 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a33533fd-fdd4-4242-a987-529067802c3c None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:00.255577 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a33533fd-fdd4-4242-a987-529067802c3c None admin] PATH_INFO: `/v3/roles` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:00.256275 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a33533fd-fdd4-4242-a987-529067802c3c None admin] RBAC: Adding query filter params (name=member) {{(pid=81578) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:43:00.256534 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a33533fd-fdd4-4242-a987-529067802c3c None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:00.258630 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a33533fd-fdd4-4242-a987-529067802c3c None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:00.271840 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 198/393] 10.222.0.27 () {56 vars in 1101 bytes} [Tue Aug 30 01:43:00 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 186 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:43:00.273026 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f224dd0-5442-49d3-a68d-16e31e9af9ae None None] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:00.273867 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f224dd0-5442-49d3-a68d-16e31e9af9ae None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:00.274143 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f224dd0-5442-49d3-a68d-16e31e9af9ae None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:00.293359 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.auth.core [None req-1f224dd0-5442-49d3-a68d-16e31e9af9ae None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81578) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 01:43:00.294643 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-1f224dd0-5442-49d3-a68d-16e31e9af9ae None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:43:00.314258 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-246156a5-008a-4234-a9d5-c12e6087451a None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:00.337846 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:43:00.359521 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0492bfba-5db0-4a26-a814-1650e7136667 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:00.360558 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0492bfba-5db0-4a26-a814-1650e7136667 None admin] REQUEST_METHOD: `PUT` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:00.361060 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0492bfba-5db0-4a26-a814-1650e7136667 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:00.361560 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0492bfba-5db0-4a26-a814-1650e7136667 None admin] PATH_INFO: `/v3/projects/90818f998d5f4520971eacf6ccc830ce/groups/40df2032c69847bbaab1d9363f127d52/roles/4621068972ba4f79bad104f956db04aa` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:00.364678 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0492bfba-5db0-4a26-a814-1650e7136667 None admin] RBAC: Authorizing `identity:create_grant(project_id=90818f998d5f4520971eacf6ccc830ce, group_id=40df2032c69847bbaab1d9363f127d52, role_id=4621068972ba4f79bad104f956db04aa)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:00.367646 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0492bfba-5db0-4a26-a814-1650e7136667 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:00.371924 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:43:00.385450 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 199/394] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Aug 30 01:43:00 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 113 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 30 01:43:00.386372 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f224dd0-5442-49d3-a68d-16e31e9af9ae None None] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:00.470333 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 196/395] 10.222.0.27 () {58 vars in 1442 bytes} [Tue Aug 30 01:43:00 2022] PUT /identity/v3/projects/90818f998d5f4520971eacf6ccc830ce/groups/40df2032c69847bbaab1d9363f127d52/roles/4621068972ba4f79bad104f956db04aa => generated 0 bytes in 284 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 01:43:00.471316 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0492bfba-5db0-4a26-a814-1650e7136667 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:00.513725 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f224dd0-5442-49d3-a68d-16e31e9af9ae None None] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:00.550725 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de07a20e-8a50-49f0-b76b-ad0afd5d2f4c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:00.551747 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de07a20e-8a50-49f0-b76b-ad0afd5d2f4c None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:00.552241 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de07a20e-8a50-49f0-b76b-ad0afd5d2f4c None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:00.552758 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de07a20e-8a50-49f0-b76b-ad0afd5d2f4c None admin] PATH_INFO: `/v3/groups/40df2032c69847bbaab1d9363f127d52` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:00.554107 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de07a20e-8a50-49f0-b76b-ad0afd5d2f4c None admin] RBAC: Authorizing `identity:get_group(group_id=40df2032c69847bbaab1d9363f127d52)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:00.556341 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de07a20e-8a50-49f0-b76b-ad0afd5d2f4c None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:00.558224 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 200/396] 10.222.0.27 () {56 vars in 1180 bytes} [Tue Aug 30 01:43:00 2022] GET /identity/v3/groups/40df2032c69847bbaab1d9363f127d52 => generated 226 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:43:00.564195 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de07a20e-8a50-49f0-b76b-ad0afd5d2f4c None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:00.593524 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0492bfba-5db0-4a26-a814-1650e7136667 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:00.618909 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:43:00.636408 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bc6182b-8bd4-4055-81ef-6515a5305ae7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:00.637406 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bc6182b-8bd4-4055-81ef-6515a5305ae7 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:00.637857 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bc6182b-8bd4-4055-81ef-6515a5305ae7 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:00.638291 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bc6182b-8bd4-4055-81ef-6515a5305ae7 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:00.650200 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8bc6182b-8bd4-4055-81ef-6515a5305ae7 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 30 01:43:00.656152 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application [None req-8bc6182b-8bd4-4055-81ef-6515a5305ae7 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Aug 30 01:43:00.656152 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 01:43:00.656152 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 01:43:00.656152 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 01:43:00.656152 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 01:43:00.656152 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 01:43:00.656152 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 01:43:00.656152 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 01:43:00.656152 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 01:43:00.656152 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 01:43:00.656152 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 01:43:00.656152 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 01:43:00.656152 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 30 01:43:00.656152 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 30 01:43:00.656152 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 30 01:43:00.656152 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 30 01:43:00.657121 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 30 01:43:00.657121 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 30 01:43:00.657121 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 30 01:43:00.657121 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 30 01:43:00.657121 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 01:43:00.657121 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 01:43:00.657121 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 01:43:00.657121 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 01:43:00.657121 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 01:43:00.657121 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 01:43:00.657121 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 01:43:00.657121 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application with Lock( Aug 30 01:43:00.657121 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 01:43:00.657121 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._enter() Aug 30 01:43:00.657121 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 01:43:00.657121 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 01:43:00.657953 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 01:43:00.657953 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.creator() Aug 30 01:43:00.657953 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 01:43:00.657953 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application created_value = creator( Aug 30 01:43:00.657953 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 30 01:43:00.657953 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 30 01:43:00.657953 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 30 01:43:00.657953 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 30 01:43:00.657953 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 30 01:43:00.657953 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 30 01:43:00.657953 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Aug 30 01:43:00.657953 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Aug 30 01:43:00.660661 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 197/397] 10.222.0.27 () {56 vars in 1099 bytes} [Tue Aug 30 01:43:00 2022] GET /identity/v3/roles/member => generated 84 bytes in 190 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 01:43:00.667648 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bc6182b-8bd4-4055-81ef-6515a5305ae7 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:00.685754 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de07a20e-8a50-49f0-b76b-ad0afd5d2f4c None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:00.732779 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31f50f2c-c537-4905-bc15-6272cadac768 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:00.733625 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31f50f2c-c537-4905-bc15-6272cadac768 None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:00.733734 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31f50f2c-c537-4905-bc15-6272cadac768 None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:00.733995 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31f50f2c-c537-4905-bc15-6272cadac768 None admin] PATH_INFO: `/v3/projects/90818f998d5f4520971eacf6ccc830ce` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:00.735298 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31f50f2c-c537-4905-bc15-6272cadac768 None admin] RBAC: Authorizing `identity:get_project(project_id=90818f998d5f4520971eacf6ccc830ce)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:00.737361 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31f50f2c-c537-4905-bc15-6272cadac768 None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:00.739777 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 201/398] 10.222.0.27 () {56 vars in 1186 bytes} [Tue Aug 30 01:43:00 2022] GET /identity/v3/projects/90818f998d5f4520971eacf6ccc830ce => generated 298 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:43:00.740643 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31f50f2c-c537-4905-bc15-6272cadac768 None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:00.744385 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:43:00.801189 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bc6182b-8bd4-4055-81ef-6515a5305ae7 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:00.836604 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3881cf6-2021-404a-a6ea-f6c1d89aceba None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:00.837221 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3881cf6-2021-404a-a6ea-f6c1d89aceba None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:00.837524 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3881cf6-2021-404a-a6ea-f6c1d89aceba None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:00.837796 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3881cf6-2021-404a-a6ea-f6c1d89aceba None admin] PATH_INFO: `/v3/roles` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:00.838460 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3881cf6-2021-404a-a6ea-f6c1d89aceba None admin] RBAC: Adding query filter params (name=member) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:43:00.838758 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3881cf6-2021-404a-a6ea-f6c1d89aceba None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:00.840142 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3881cf6-2021-404a-a6ea-f6c1d89aceba None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:00.848779 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 198/399] 10.222.0.27 () {56 vars in 1101 bytes} [Tue Aug 30 01:43:00 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:43:00.849680 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3881cf6-2021-404a-a6ea-f6c1d89aceba None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:00.868718 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31f50f2c-c537-4905-bc15-6272cadac768 None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:00.885805 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:43:00.908350 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-232e67e5-3487-4abf-9024-287fe18a845d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:00.910233 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-232e67e5-3487-4abf-9024-287fe18a845d None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:00.910233 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-232e67e5-3487-4abf-9024-287fe18a845d None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:00.910233 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-232e67e5-3487-4abf-9024-287fe18a845d None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:00.930911 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-232e67e5-3487-4abf-9024-287fe18a845d None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 30 01:43:00.938132 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application [None req-232e67e5-3487-4abf-9024-287fe18a845d None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Aug 30 01:43:00.938132 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 01:43:00.938132 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 01:43:00.938132 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 01:43:00.938132 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 01:43:00.938132 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 01:43:00.938132 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 01:43:00.938132 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 01:43:00.938132 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 01:43:00.938132 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 01:43:00.938132 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 01:43:00.938132 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 01:43:00.938132 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 30 01:43:00.938132 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 30 01:43:00.938132 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 30 01:43:00.938132 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 30 01:43:00.939086 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 30 01:43:00.939086 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 30 01:43:00.939086 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 30 01:43:00.939086 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 30 01:43:00.939086 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 01:43:00.939086 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 01:43:00.939086 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 01:43:00.939086 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 01:43:00.939086 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 01:43:00.939086 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 01:43:00.939086 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 01:43:00.939086 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application with Lock( Aug 30 01:43:00.939086 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 01:43:00.939086 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._enter() Aug 30 01:43:00.939086 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 01:43:00.939086 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 01:43:00.939882 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 01:43:00.939882 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.creator() Aug 30 01:43:00.939882 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 01:43:00.939882 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application created_value = creator( Aug 30 01:43:00.939882 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 30 01:43:00.939882 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 30 01:43:00.939882 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 30 01:43:00.939882 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 30 01:43:00.939882 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 30 01:43:00.939882 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 30 01:43:00.939882 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Aug 30 01:43:00.939882 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application Aug 30 01:43:00.940486 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 202/400] 10.222.0.27 () {56 vars in 1114 bytes} [Tue Aug 30 01:43:00 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 200 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 01:43:00.941027 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-232e67e5-3487-4abf-9024-287fe18a845d None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:00.976096 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3881cf6-2021-404a-a6ea-f6c1d89aceba None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:00.996442 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:43:01.013995 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2eca5323-9e23-4117-a833-4d95c7f45966 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:01.014706 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2eca5323-9e23-4117-a833-4d95c7f45966 None admin] REQUEST_METHOD: `PUT` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:01.014981 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2eca5323-9e23-4117-a833-4d95c7f45966 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:01.015247 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2eca5323-9e23-4117-a833-4d95c7f45966 None admin] PATH_INFO: `/v3/projects/90818f998d5f4520971eacf6ccc830ce/groups/40df2032c69847bbaab1d9363f127d52/roles/1fb81afcb9f245f3b989add94598a063` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:01.017365 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2eca5323-9e23-4117-a833-4d95c7f45966 None admin] RBAC: Authorizing `identity:create_grant(project_id=90818f998d5f4520971eacf6ccc830ce, group_id=40df2032c69847bbaab1d9363f127d52, role_id=1fb81afcb9f245f3b989add94598a063)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:01.018887 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2eca5323-9e23-4117-a833-4d95c7f45966 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:01.081657 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 199/401] 10.222.0.27 () {58 vars in 1442 bytes} [Tue Aug 30 01:43:00 2022] PUT /identity/v3/projects/90818f998d5f4520971eacf6ccc830ce/groups/40df2032c69847bbaab1d9363f127d52/roles/1fb81afcb9f245f3b989add94598a063 => generated 0 bytes in 231 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 01:43:01.082911 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2eca5323-9e23-4117-a833-4d95c7f45966 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:01.090880 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-232e67e5-3487-4abf-9024-287fe18a845d None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:01.131759 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04d2f14e-a2fd-428e-a3db-244a5d709f67 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:01.132990 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04d2f14e-a2fd-428e-a3db-244a5d709f67 None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:01.133512 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04d2f14e-a2fd-428e-a3db-244a5d709f67 None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:01.133980 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04d2f14e-a2fd-428e-a3db-244a5d709f67 None admin] PATH_INFO: `/v3/groups/40df2032c69847bbaab1d9363f127d52` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:01.135320 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04d2f14e-a2fd-428e-a3db-244a5d709f67 None admin] RBAC: Authorizing `identity:get_group(group_id=40df2032c69847bbaab1d9363f127d52)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:01.137058 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04d2f14e-a2fd-428e-a3db-244a5d709f67 None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:01.139341 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 203/402] 10.222.0.27 () {56 vars in 1180 bytes} [Tue Aug 30 01:43:00 2022] GET /identity/v3/groups/40df2032c69847bbaab1d9363f127d52 => generated 226 bytes in 199 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:43:01.150796 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04d2f14e-a2fd-428e-a3db-244a5d709f67 None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:01.217366 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:43:01.225002 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2eca5323-9e23-4117-a833-4d95c7f45966 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:01.273700 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14bf50ff-4954-46a7-a000-df9894217107 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:01.274716 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14bf50ff-4954-46a7-a000-df9894217107 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:01.275070 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14bf50ff-4954-46a7-a000-df9894217107 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:01.275349 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14bf50ff-4954-46a7-a000-df9894217107 None admin] PATH_INFO: `/v3/roles` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:01.276034 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14bf50ff-4954-46a7-a000-df9894217107 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:43:01.276343 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14bf50ff-4954-46a7-a000-df9894217107 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:01.277893 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14bf50ff-4954-46a7-a000-df9894217107 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:01.287290 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 200/403] 10.222.0.27 () {56 vars in 1111 bytes} [Tue Aug 30 01:43:01 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 205 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:43:01.288559 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1436e9a-40da-42ac-aaea-b839e6f86e91 None None] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:01.289520 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1436e9a-40da-42ac-aaea-b839e6f86e91 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:01.289942 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1436e9a-40da-42ac-aaea-b839e6f86e91 None None] PATH_INFO: `/` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:01.291865 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 201/404] 10.222.0.27 () {54 vars in 927 bytes} [Tue Aug 30 01:43:01 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 01:43:01.293338 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1436e9a-40da-42ac-aaea-b839e6f86e91 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:01.303423 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04d2f14e-a2fd-428e-a3db-244a5d709f67 None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:01.349168 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:43:01.357596 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9f23651-5e40-446a-9628-98cd274284eb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:01.358241 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9f23651-5e40-446a-9628-98cd274284eb None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:01.365986 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9f23651-5e40-446a-9628-98cd274284eb None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:01.365986 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9f23651-5e40-446a-9628-98cd274284eb None admin] PATH_INFO: `/v3/projects/67913d5682a443069a5eed96f179751f` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:01.365986 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9f23651-5e40-446a-9628-98cd274284eb None admin] RBAC: Authorizing `identity:get_project(project_id=67913d5682a443069a5eed96f179751f)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:01.365986 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9f23651-5e40-446a-9628-98cd274284eb None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:01.365986 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 204/405] 10.222.0.27 () {56 vars in 1186 bytes} [Tue Aug 30 01:43:01 2022] GET /identity/v3/projects/67913d5682a443069a5eed96f179751f => generated 302 bytes in 213 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:43:01.365986 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8e574cc-0c1b-43a7-8440-3efbf4ec1873 None None] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:01.366571 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8e574cc-0c1b-43a7-8440-3efbf4ec1873 None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:01.366571 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8e574cc-0c1b-43a7-8440-3efbf4ec1873 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:01.388679 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.auth.core [None req-b8e574cc-0c1b-43a7-8440-3efbf4ec1873 None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81578) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 01:43:01.389930 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-b8e574cc-0c1b-43a7-8440-3efbf4ec1873 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:43:01.433985 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1436e9a-40da-42ac-aaea-b839e6f86e91 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:01.465903 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 205/406] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Aug 30 01:43:01 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 30 01:43:01.466754 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8e574cc-0c1b-43a7-8440-3efbf4ec1873 None None] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:01.489028 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14b3996d-bdb2-420f-b489-3162e47c8365 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:01.489991 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14b3996d-bdb2-420f-b489-3162e47c8365 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:01.490643 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14b3996d-bdb2-420f-b489-3162e47c8365 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:01.491066 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14b3996d-bdb2-420f-b489-3162e47c8365 None admin] PATH_INFO: `/v3/groups/40df2032c69847bbaab1d9363f127d52` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:01.492790 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14b3996d-bdb2-420f-b489-3162e47c8365 None admin] RBAC: Authorizing `identity:get_group(group_id=40df2032c69847bbaab1d9363f127d52)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:01.494698 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14b3996d-bdb2-420f-b489-3162e47c8365 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:01.496874 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 202/407] 10.222.0.27 () {56 vars in 1180 bytes} [Tue Aug 30 01:43:01 2022] GET /identity/v3/groups/40df2032c69847bbaab1d9363f127d52 => generated 226 bytes in 204 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:43:01.502516 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14b3996d-bdb2-420f-b489-3162e47c8365 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:01.559487 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:43:01.629654 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14b3996d-bdb2-420f-b489-3162e47c8365 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:01.640527 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:43:01.668126 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b7bcc16-0f08-49c5-ae87-8f034e2ec3ad None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:01.669171 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b7bcc16-0f08-49c5-ae87-8f034e2ec3ad None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:01.669792 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b7bcc16-0f08-49c5-ae87-8f034e2ec3ad None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:01.670287 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b7bcc16-0f08-49c5-ae87-8f034e2ec3ad None admin] PATH_INFO: `/v3/projects/67913d5682a443069a5eed96f179751f` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:01.671642 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b7bcc16-0f08-49c5-ae87-8f034e2ec3ad None admin] RBAC: Authorizing `identity:get_project(project_id=67913d5682a443069a5eed96f179751f)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:01.673299 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b7bcc16-0f08-49c5-ae87-8f034e2ec3ad None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:01.675518 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 203/408] 10.222.0.27 () {56 vars in 1186 bytes} [Tue Aug 30 01:43:01 2022] GET /identity/v3/projects/67913d5682a443069a5eed96f179751f => generated 302 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 30 01:43:01.681650 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b7bcc16-0f08-49c5-ae87-8f034e2ec3ad None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:01.697052 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8e574cc-0c1b-43a7-8440-3efbf4ec1873 None None] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:01.769557 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdb5c708-59ce-4e0f-918b-da555a4614f1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:01.770226 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdb5c708-59ce-4e0f-918b-da555a4614f1 None admin] REQUEST_METHOD: `PUT` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:01.770488 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdb5c708-59ce-4e0f-918b-da555a4614f1 None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:01.771481 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdb5c708-59ce-4e0f-918b-da555a4614f1 None admin] PATH_INFO: `/v3/projects/67913d5682a443069a5eed96f179751f/groups/40df2032c69847bbaab1d9363f127d52/roles/1fb81afcb9f245f3b989add94598a063` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:01.775140 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bdb5c708-59ce-4e0f-918b-da555a4614f1 None admin] RBAC: Authorizing `identity:create_grant(project_id=67913d5682a443069a5eed96f179751f, group_id=40df2032c69847bbaab1d9363f127d52, role_id=1fb81afcb9f245f3b989add94598a063)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:01.776806 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bdb5c708-59ce-4e0f-918b-da555a4614f1 None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:01.848006 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b7bcc16-0f08-49c5-ae87-8f034e2ec3ad None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:01.874134 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 206/409] 10.222.0.27 () {58 vars in 1442 bytes} [Tue Aug 30 01:43:01 2022] PUT /identity/v3/projects/67913d5682a443069a5eed96f179751f/groups/40df2032c69847bbaab1d9363f127d52/roles/1fb81afcb9f245f3b989add94598a063 => generated 0 bytes in 402 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 01:43:01.875290 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdb5c708-59ce-4e0f-918b-da555a4614f1 None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:01.904755 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1314dcc5-bedc-43dc-8dbf-fe785ae7dedc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:01.905782 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1314dcc5-bedc-43dc-8dbf-fe785ae7dedc None admin] REQUEST_METHOD: `PUT` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:01.906117 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1314dcc5-bedc-43dc-8dbf-fe785ae7dedc None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:01.906386 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1314dcc5-bedc-43dc-8dbf-fe785ae7dedc None admin] PATH_INFO: `/v3/projects/67913d5682a443069a5eed96f179751f/groups/40df2032c69847bbaab1d9363f127d52/roles/4621068972ba4f79bad104f956db04aa` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:01.909783 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1314dcc5-bedc-43dc-8dbf-fe785ae7dedc None admin] RBAC: Authorizing `identity:create_grant(project_id=67913d5682a443069a5eed96f179751f, group_id=40df2032c69847bbaab1d9363f127d52, role_id=4621068972ba4f79bad104f956db04aa)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:01.911636 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1314dcc5-bedc-43dc-8dbf-fe785ae7dedc None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:01.995211 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 204/410] 10.222.0.27 () {58 vars in 1442 bytes} [Tue Aug 30 01:43:01 2022] PUT /identity/v3/projects/67913d5682a443069a5eed96f179751f/groups/40df2032c69847bbaab1d9363f127d52/roles/4621068972ba4f79bad104f956db04aa => generated 0 bytes in 314 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 01:43:02.029246 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:43:02.035960 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdb5c708-59ce-4e0f-918b-da555a4614f1 None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:02.079711 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6530913b-0a30-4471-97a9-237e7a21b06d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:02.080506 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6530913b-0a30-4471-97a9-237e7a21b06d None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:02.080850 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6530913b-0a30-4471-97a9-237e7a21b06d None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:02.081502 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6530913b-0a30-4471-97a9-237e7a21b06d None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:02.094531 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6530913b-0a30-4471-97a9-237e7a21b06d None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 30 01:43:02.099732 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application [None req-6530913b-0a30-4471-97a9-237e7a21b06d None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 30 01:43:02.099732 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 01:43:02.099732 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 01:43:02.099732 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 01:43:02.099732 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 01:43:02.099732 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 01:43:02.099732 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 01:43:02.099732 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 01:43:02.099732 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 01:43:02.099732 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 01:43:02.099732 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 01:43:02.099732 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 01:43:02.099732 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 30 01:43:02.099732 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 30 01:43:02.099732 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 30 01:43:02.099732 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 30 01:43:02.102519 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 30 01:43:02.102519 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 30 01:43:02.102519 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 30 01:43:02.102519 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 30 01:43:02.102519 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 01:43:02.102519 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 01:43:02.102519 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 01:43:02.102519 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 01:43:02.102519 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 01:43:02.102519 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 01:43:02.102519 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 01:43:02.102519 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application with Lock( Aug 30 01:43:02.102519 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 01:43:02.102519 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._enter() Aug 30 01:43:02.102519 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 01:43:02.102519 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 01:43:02.104526 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 01:43:02.104526 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.creator() Aug 30 01:43:02.104526 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 01:43:02.104526 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application created_value = creator( Aug 30 01:43:02.104526 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 30 01:43:02.104526 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 30 01:43:02.104526 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 30 01:43:02.104526 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 30 01:43:02.104526 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 30 01:43:02.104526 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 30 01:43:02.104526 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 30 01:43:02.104526 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application Aug 30 01:43:02.104526 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 207/411] 10.222.0.27 () {56 vars in 1096 bytes} [Tue Aug 30 01:43:01 2022] GET /identity/v3/roles/admin => generated 83 bytes in 226 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 01:43:02.114790 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1314dcc5-bedc-43dc-8dbf-fe785ae7dedc None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:02.161311 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:43:02.237774 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1314dcc5-bedc-43dc-8dbf-fe785ae7dedc None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:02.275712 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd4e4c40-528e-4a50-9d21-28ba48b34384 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:02.276617 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd4e4c40-528e-4a50-9d21-28ba48b34384 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:02.277122 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd4e4c40-528e-4a50-9d21-28ba48b34384 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:02.280207 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd4e4c40-528e-4a50-9d21-28ba48b34384 None admin] PATH_INFO: `/v3/roles` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:02.280911 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd4e4c40-528e-4a50-9d21-28ba48b34384 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:43:02.281272 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd4e4c40-528e-4a50-9d21-28ba48b34384 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:02.282797 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd4e4c40-528e-4a50-9d21-28ba48b34384 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:02.292164 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 205/412] 10.222.0.27 () {56 vars in 1099 bytes} [Tue Aug 30 01:43:02 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:43:02.297653 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6530913b-0a30-4471-97a9-237e7a21b06d None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:02.457480 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6530913b-0a30-4471-97a9-237e7a21b06d None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:02.500883 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c119e218-6859-49d5-a486-6e6700633ff4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:02.501941 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c119e218-6859-49d5-a486-6e6700633ff4 None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:02.502438 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c119e218-6859-49d5-a486-6e6700633ff4 None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:02.502919 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c119e218-6859-49d5-a486-6e6700633ff4 None admin] PATH_INFO: `/v3/groups/e130c65bd71a4dc983e47fda13b8d556` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:02.504244 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c119e218-6859-49d5-a486-6e6700633ff4 None admin] RBAC: Authorizing `identity:get_group(group_id=e130c65bd71a4dc983e47fda13b8d556)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:02.505978 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c119e218-6859-49d5-a486-6e6700633ff4 None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:02.508197 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 208/413] 10.222.0.27 () {56 vars in 1180 bytes} [Tue Aug 30 01:43:02 2022] GET /identity/v3/groups/e130c65bd71a4dc983e47fda13b8d556 => generated 229 bytes in 211 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 30 01:43:02.513988 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd4e4c40-528e-4a50-9d21-28ba48b34384 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:02.651978 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:43:02.657920 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd4e4c40-528e-4a50-9d21-28ba48b34384 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:02.699367 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73b651bc-c5ee-44e8-b6db-e0b1dc741f64 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:02.700008 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73b651bc-c5ee-44e8-b6db-e0b1dc741f64 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:02.700287 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73b651bc-c5ee-44e8-b6db-e0b1dc741f64 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:02.700557 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73b651bc-c5ee-44e8-b6db-e0b1dc741f64 None admin] PATH_INFO: `/v3/projects/70bc9f42895844308fb4b9192314f80b` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:02.701710 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73b651bc-c5ee-44e8-b6db-e0b1dc741f64 None admin] RBAC: Authorizing `identity:get_project(project_id=70bc9f42895844308fb4b9192314f80b)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:02.703174 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73b651bc-c5ee-44e8-b6db-e0b1dc741f64 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:02.711854 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 206/414] 10.222.0.27 () {56 vars in 1186 bytes} [Tue Aug 30 01:43:02 2022] GET /identity/v3/projects/70bc9f42895844308fb4b9192314f80b => generated 344 bytes in 192 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:43:02.712079 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c119e218-6859-49d5-a486-6e6700633ff4 None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:02.753660 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:43:02.865765 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c119e218-6859-49d5-a486-6e6700633ff4 None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:02.919476 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19572f3e-7ff4-4873-9118-5c7c90bff6ff None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:02.920295 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19572f3e-7ff4-4873-9118-5c7c90bff6ff None admin] REQUEST_METHOD: `PUT` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:02.920575 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19572f3e-7ff4-4873-9118-5c7c90bff6ff None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:02.920843 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19572f3e-7ff4-4873-9118-5c7c90bff6ff None admin] PATH_INFO: `/v3/projects/70bc9f42895844308fb4b9192314f80b/groups/e130c65bd71a4dc983e47fda13b8d556/roles/ead5fac86a0747ea8823a82efcea8ed7` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:02.922783 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19572f3e-7ff4-4873-9118-5c7c90bff6ff None admin] RBAC: Authorizing `identity:create_grant(project_id=70bc9f42895844308fb4b9192314f80b, group_id=e130c65bd71a4dc983e47fda13b8d556, role_id=ead5fac86a0747ea8823a82efcea8ed7)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:02.924197 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19572f3e-7ff4-4873-9118-5c7c90bff6ff None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:02.944632 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4205c717-3101-490e-b639-8cc5fe5a2de1 None None] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:02.945853 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4205c717-3101-490e-b639-8cc5fe5a2de1 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:02.946348 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4205c717-3101-490e-b639-8cc5fe5a2de1 None None] PATH_INFO: `/` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:02.948209 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 207/415] 10.222.0.27 () {54 vars in 927 bytes} [Tue Aug 30 01:43:02 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 01:43:02.954476 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba1ded68-7ceb-4bf9-8549-c76ffc5b7aa8 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:02.954979 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba1ded68-7ceb-4bf9-8549-c76ffc5b7aa8 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:02.955454 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba1ded68-7ceb-4bf9-8549-c76ffc5b7aa8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:02.974989 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-ba1ded68-7ceb-4bf9-8549-c76ffc5b7aa8 None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 01:43:02.977040 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-ba1ded68-7ceb-4bf9-8549-c76ffc5b7aa8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:43:02.999811 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 209/416] 10.222.0.27 () {58 vars in 1442 bytes} [Tue Aug 30 01:43:02 2022] PUT /identity/v3/projects/70bc9f42895844308fb4b9192314f80b/groups/e130c65bd71a4dc983e47fda13b8d556/roles/ead5fac86a0747ea8823a82efcea8ed7 => generated 0 bytes in 289 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 01:43:03.078429 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 208/417] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Aug 30 01:43:02 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 125 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Aug 30 01:43:03.462446 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19572f3e-7ff4-4873-9118-5c7c90bff6ff None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:03.534812 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba66b5c7-02e8-4985-85df-e966f16f950c None None] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:03.535463 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba66b5c7-02e8-4985-85df-e966f16f950c None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:03.535924 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba66b5c7-02e8-4985-85df-e966f16f950c None None] PATH_INFO: `/` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:03.539552 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 209/418] 10.222.0.27 () {54 vars in 927 bytes} [Tue Aug 30 01:43:03 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 01:43:03.544116 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a871bb5d-c64c-453b-a0e8-e77819efeaae None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:03.544642 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a871bb5d-c64c-453b-a0e8-e77819efeaae None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:03.545119 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a871bb5d-c64c-453b-a0e8-e77819efeaae None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:03.560659 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:43:03.585077 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-a871bb5d-c64c-453b-a0e8-e77819efeaae None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 01:43:03.588228 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-a871bb5d-c64c-453b-a0e8-e77819efeaae None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:43:03.593132 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19572f3e-7ff4-4873-9118-5c7c90bff6ff None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:03.658887 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4417f015-69a8-4327-8f09-0ed07d79dfe4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:03.659898 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4417f015-69a8-4327-8f09-0ed07d79dfe4 None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:03.661194 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4417f015-69a8-4327-8f09-0ed07d79dfe4 None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:03.661685 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4417f015-69a8-4327-8f09-0ed07d79dfe4 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:03.682808 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4417f015-69a8-4327-8f09-0ed07d79dfe4 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 30 01:43:03.689578 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application [None req-4417f015-69a8-4327-8f09-0ed07d79dfe4 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Aug 30 01:43:03.689578 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 01:43:03.689578 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 01:43:03.689578 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 01:43:03.689578 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 01:43:03.689578 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 01:43:03.689578 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 01:43:03.689578 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 01:43:03.689578 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 01:43:03.689578 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 01:43:03.689578 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 01:43:03.689578 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 01:43:03.689578 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 30 01:43:03.689578 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 30 01:43:03.689578 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 30 01:43:03.689578 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 30 01:43:03.690550 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 30 01:43:03.690550 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 30 01:43:03.690550 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 30 01:43:03.690550 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 30 01:43:03.690550 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 01:43:03.690550 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 01:43:03.690550 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 01:43:03.690550 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 01:43:03.690550 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 01:43:03.690550 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 01:43:03.690550 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 01:43:03.690550 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application with Lock( Aug 30 01:43:03.690550 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 01:43:03.690550 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._enter() Aug 30 01:43:03.690550 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 01:43:03.690550 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 01:43:03.691382 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 01:43:03.691382 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.creator() Aug 30 01:43:03.691382 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 01:43:03.691382 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application created_value = creator( Aug 30 01:43:03.691382 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 30 01:43:03.691382 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 30 01:43:03.691382 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 30 01:43:03.691382 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 30 01:43:03.691382 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 30 01:43:03.691382 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 30 01:43:03.691382 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Aug 30 01:43:03.691382 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application Aug 30 01:43:03.698939 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 210/419] 10.222.0.27 () {56 vars in 1114 bytes} [Tue Aug 30 01:43:03 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 236 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 01:43:03.706506 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4417f015-69a8-4327-8f09-0ed07d79dfe4 None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:03.706789 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 210/420] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Aug 30 01:43:03 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 158 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Aug 30 01:43:03.819391 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4417f015-69a8-4327-8f09-0ed07d79dfe4 None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:03.862186 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b01ecfe-8bb3-4778-bb8d-9a744af6f0b8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:03.862806 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b01ecfe-8bb3-4778-bb8d-9a744af6f0b8 None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:03.863088 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b01ecfe-8bb3-4778-bb8d-9a744af6f0b8 None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:03.863356 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b01ecfe-8bb3-4778-bb8d-9a744af6f0b8 None admin] PATH_INFO: `/v3/roles` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:03.864022 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b01ecfe-8bb3-4778-bb8d-9a744af6f0b8 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81578) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:43:03.864324 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b01ecfe-8bb3-4778-bb8d-9a744af6f0b8 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:03.865702 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b01ecfe-8bb3-4778-bb8d-9a744af6f0b8 None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:03.874296 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 211/421] 10.222.0.27 () {56 vars in 1111 bytes} [Tue Aug 30 01:43:03 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:43:03.879848 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a871bb5d-c64c-453b-a0e8-e77819efeaae None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:03.921077 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:43:04.015043 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a871bb5d-c64c-453b-a0e8-e77819efeaae None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:04.062857 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ecc80cf-9a65-43fd-98ed-d75e162b4f84 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:04.063476 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ecc80cf-9a65-43fd-98ed-d75e162b4f84 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:04.063797 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ecc80cf-9a65-43fd-98ed-d75e162b4f84 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:04.064014 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ecc80cf-9a65-43fd-98ed-d75e162b4f84 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:04.119584 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b01ecfe-8bb3-4778-bb8d-9a744af6f0b8 None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:04.129126 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:43:04.171514 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ecc80cf-9a65-43fd-98ed-d75e162b4f84 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:04.172982 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ecc80cf-9a65-43fd-98ed-d75e162b4f84 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:04.227730 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:43:04.282261 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b01ecfe-8bb3-4778-bb8d-9a744af6f0b8 None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:04.312510 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 211/422] 10.222.0.27 () {58 vars in 1282 bytes} [Tue Aug 30 01:43:03 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 433 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 30 01:43:04.314262 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56a70ba4-7398-4f9e-9c5c-69d297cb259f None None] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:04.314763 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56a70ba4-7398-4f9e-9c5c-69d297cb259f None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:04.315098 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56a70ba4-7398-4f9e-9c5c-69d297cb259f None None] PATH_INFO: `/` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:04.316921 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 212/423] 10.222.0.27 () {54 vars in 927 bytes} [Tue Aug 30 01:43:04 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 01:43:04.322464 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56a70ba4-7398-4f9e-9c5c-69d297cb259f None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:04.325661 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ea412b1-7281-42f9-858f-c2bcb57b48d7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:04.326304 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ea412b1-7281-42f9-858f-c2bcb57b48d7 None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:04.326582 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ea412b1-7281-42f9-858f-c2bcb57b48d7 None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:04.326852 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ea412b1-7281-42f9-858f-c2bcb57b48d7 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:04.337655 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ea412b1-7281-42f9-858f-c2bcb57b48d7 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 30 01:43:04.343031 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application [None req-8ea412b1-7281-42f9-858f-c2bcb57b48d7 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Aug 30 01:43:04.343031 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 01:43:04.343031 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 01:43:04.343031 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 01:43:04.343031 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 01:43:04.343031 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 01:43:04.343031 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 01:43:04.343031 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 01:43:04.343031 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 01:43:04.343031 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 01:43:04.343031 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 01:43:04.343031 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 01:43:04.343031 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 30 01:43:04.343031 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 30 01:43:04.343031 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 30 01:43:04.343031 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 30 01:43:04.343983 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 30 01:43:04.343983 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 30 01:43:04.343983 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 30 01:43:04.343983 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 30 01:43:04.343983 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 01:43:04.343983 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 01:43:04.343983 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 01:43:04.343983 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 01:43:04.343983 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 01:43:04.343983 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 01:43:04.343983 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 01:43:04.343983 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application with Lock( Aug 30 01:43:04.343983 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 01:43:04.343983 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._enter() Aug 30 01:43:04.343983 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 01:43:04.343983 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 01:43:04.344780 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 01:43:04.344780 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.creator() Aug 30 01:43:04.344780 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 01:43:04.344780 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application created_value = creator( Aug 30 01:43:04.344780 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 30 01:43:04.344780 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 30 01:43:04.344780 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 30 01:43:04.344780 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 30 01:43:04.344780 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 30 01:43:04.344780 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 30 01:43:04.344780 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Aug 30 01:43:04.344780 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application Aug 30 01:43:04.344780 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 212/424] 10.222.0.27 () {56 vars in 1099 bytes} [Tue Aug 30 01:43:04 2022] GET /identity/v3/roles/member => generated 84 bytes in 225 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 01:43:04.345966 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da1e1f80-9c99-43db-989e-4e1b2685f572 None None] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:04.345966 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da1e1f80-9c99-43db-989e-4e1b2685f572 None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:04.346140 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da1e1f80-9c99-43db-989e-4e1b2685f572 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:04.364973 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.auth.core [None req-da1e1f80-9c99-43db-989e-4e1b2685f572 None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81578) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 01:43:04.366228 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-da1e1f80-9c99-43db-989e-4e1b2685f572 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:43:04.432029 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:43:04.436447 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56a70ba4-7398-4f9e-9c5c-69d297cb259f None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:04.440798 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 213/425] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Aug 30 01:43:04 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 30 01:43:04.441666 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da1e1f80-9c99-43db-989e-4e1b2685f572 None None] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:04.474153 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:43:04.484508 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50e7587b-0ea4-4ad3-8b9d-bdc27fc99d16 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:04.492684 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50e7587b-0ea4-4ad3-8b9d-bdc27fc99d16 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:04.492684 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50e7587b-0ea4-4ad3-8b9d-bdc27fc99d16 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:04.492684 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50e7587b-0ea4-4ad3-8b9d-bdc27fc99d16 None admin] PATH_INFO: `/v3/projects/90818f998d5f4520971eacf6ccc830ce` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:04.492684 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50e7587b-0ea4-4ad3-8b9d-bdc27fc99d16 None admin] RBAC: Authorizing `identity:get_project(project_id=90818f998d5f4520971eacf6ccc830ce)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:04.492684 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50e7587b-0ea4-4ad3-8b9d-bdc27fc99d16 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:04.492684 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 213/426] 10.222.0.27 () {56 vars in 1186 bytes} [Tue Aug 30 01:43:04 2022] GET /identity/v3/projects/90818f998d5f4520971eacf6ccc830ce => generated 298 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:43:04.493184 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c811ba6-adfd-4fa7-b99e-e397c895d5e5 None None] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:04.493184 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c811ba6-adfd-4fa7-b99e-e397c895d5e5 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:04.493184 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c811ba6-adfd-4fa7-b99e-e397c895d5e5 None None] PATH_INFO: `/` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:04.493416 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 214/427] 10.222.0.27 () {54 vars in 927 bytes} [Tue Aug 30 01:43:04 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 01:43:04.495774 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c811ba6-adfd-4fa7-b99e-e397c895d5e5 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:04.580319 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da1e1f80-9c99-43db-989e-4e1b2685f572 None None] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:04.611317 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c811ba6-adfd-4fa7-b99e-e397c895d5e5 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:04.626297 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-beb8bb3d-0149-406e-94b0-464d8faaadde None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:04.627231 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-beb8bb3d-0149-406e-94b0-464d8faaadde None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:04.627592 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-beb8bb3d-0149-406e-94b0-464d8faaadde None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:04.627916 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-beb8bb3d-0149-406e-94b0-464d8faaadde None admin] PATH_INFO: `/v3/roles` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:04.628636 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-beb8bb3d-0149-406e-94b0-464d8faaadde None admin] RBAC: Adding query filter params (name=member) {{(pid=81578) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:43:04.628939 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-beb8bb3d-0149-406e-94b0-464d8faaadde None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:04.630443 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-beb8bb3d-0149-406e-94b0-464d8faaadde None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:04.641109 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 214/428] 10.222.0.27 () {56 vars in 1101 bytes} [Tue Aug 30 01:43:04 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 199 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:43:04.642421 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cc5ba38-4856-43a6-94ae-9d1aef156c26 None None] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:04.642760 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cc5ba38-4856-43a6-94ae-9d1aef156c26 None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:04.643042 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cc5ba38-4856-43a6-94ae-9d1aef156c26 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:04.655151 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-132fa499-22b3-4f83-ac14-bebde323d08b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:04.655789 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-132fa499-22b3-4f83-ac14-bebde323d08b None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:04.656070 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-132fa499-22b3-4f83-ac14-bebde323d08b None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:04.656351 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-132fa499-22b3-4f83-ac14-bebde323d08b None admin] PATH_INFO: `/v3/groups/40df2032c69847bbaab1d9363f127d52` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:04.657488 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-132fa499-22b3-4f83-ac14-bebde323d08b None admin] RBAC: Authorizing `identity:get_group(group_id=40df2032c69847bbaab1d9363f127d52)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:04.658988 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-132fa499-22b3-4f83-ac14-bebde323d08b None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:04.660859 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 215/429] 10.222.0.27 () {56 vars in 1180 bytes} [Tue Aug 30 01:43:04 2022] GET /identity/v3/groups/40df2032c69847bbaab1d9363f127d52 => generated 226 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 30 01:43:04.661744 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-132fa499-22b3-4f83-ac14-bebde323d08b None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:04.663243 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.auth.core [None req-1cc5ba38-4856-43a6-94ae-9d1aef156c26 None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81578) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 01:43:04.664896 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-1cc5ba38-4856-43a6-94ae-9d1aef156c26 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:43:04.725254 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:43:04.726657 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:43:04.748970 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 215/430] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Aug 30 01:43:04 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 30 01:43:04.750526 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cc5ba38-4856-43a6-94ae-9d1aef156c26 None None] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:04.771396 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-132fa499-22b3-4f83-ac14-bebde323d08b None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:04.818425 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c35f01eb-6de0-423f-8e95-0db8a44aedbc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:04.819485 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c35f01eb-6de0-423f-8e95-0db8a44aedbc None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:04.820605 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c35f01eb-6de0-423f-8e95-0db8a44aedbc None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:04.821232 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c35f01eb-6de0-423f-8e95-0db8a44aedbc None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:04.865715 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cc5ba38-4856-43a6-94ae-9d1aef156c26 None None] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:04.900313 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c35f01eb-6de0-423f-8e95-0db8a44aedbc None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:04.901991 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c35f01eb-6de0-423f-8e95-0db8a44aedbc None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:04.902539 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c59a4fe4-7917-410f-9e9e-2529d828e552 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:04.903633 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c59a4fe4-7917-410f-9e9e-2529d828e552 None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:04.904044 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c59a4fe4-7917-410f-9e9e-2529d828e552 None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:04.904483 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c59a4fe4-7917-410f-9e9e-2529d828e552 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:04.905370 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c59a4fe4-7917-410f-9e9e-2529d828e552 None admin] RBAC: Adding query filter params (group.id=40df2032c69847bbaab1d9363f127d52, role.id=4621068972ba4f79bad104f956db04aa, scope.project.id=90818f998d5f4520971eacf6ccc830ce) {{(pid=81578) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:43:04.905942 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c59a4fe4-7917-410f-9e9e-2529d828e552 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:04.907692 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c59a4fe4-7917-410f-9e9e-2529d828e552 None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:04.914432 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 216/431] 10.222.0.27 () {56 vars in 1376 bytes} [Tue Aug 30 01:43:04 2022] GET /identity/v3/role_assignments?group.id=40df2032c69847bbaab1d9363f127d52&scope.project.id=90818f998d5f4520971eacf6ccc830ce&role.id=4621068972ba4f79bad104f956db04aa => generated 611 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:43:04.915545 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c59a4fe4-7917-410f-9e9e-2529d828e552 None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:04.974432 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:43:04.976652 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:43:05.022509 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 216/432] 10.222.0.27 () {58 vars in 1282 bytes} [Tue Aug 30 01:43:04 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 361 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 30 01:43:05.028056 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c59a4fe4-7917-410f-9e9e-2529d828e552 None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:05.028649 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c35f01eb-6de0-423f-8e95-0db8a44aedbc None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:05.066992 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9043ef2-a332-4795-ba2c-5badcae74698 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:05.068047 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9043ef2-a332-4795-ba2c-5badcae74698 None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:05.068532 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9043ef2-a332-4795-ba2c-5badcae74698 None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:05.069060 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9043ef2-a332-4795-ba2c-5badcae74698 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:05.082297 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9043ef2-a332-4795-ba2c-5badcae74698 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 30 01:43:05.089352 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application [None req-c9043ef2-a332-4795-ba2c-5badcae74698 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Aug 30 01:43:05.089352 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 01:43:05.089352 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 01:43:05.089352 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 01:43:05.089352 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 01:43:05.089352 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 01:43:05.089352 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 01:43:05.089352 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 01:43:05.089352 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 01:43:05.089352 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 01:43:05.089352 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 01:43:05.089352 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 01:43:05.089352 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 30 01:43:05.089352 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 30 01:43:05.089352 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 30 01:43:05.089352 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 30 01:43:05.090338 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 30 01:43:05.090338 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 30 01:43:05.090338 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 30 01:43:05.090338 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 30 01:43:05.090338 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 01:43:05.090338 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 01:43:05.090338 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 01:43:05.090338 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 01:43:05.090338 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 01:43:05.090338 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 01:43:05.090338 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 01:43:05.090338 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application with Lock( Aug 30 01:43:05.090338 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 01:43:05.090338 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._enter() Aug 30 01:43:05.090338 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 01:43:05.090338 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 01:43:05.091135 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 01:43:05.091135 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.creator() Aug 30 01:43:05.091135 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 01:43:05.091135 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application created_value = creator( Aug 30 01:43:05.091135 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 30 01:43:05.091135 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 30 01:43:05.091135 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 30 01:43:05.091135 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 30 01:43:05.091135 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 30 01:43:05.091135 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 30 01:43:05.091135 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Aug 30 01:43:05.091135 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application Aug 30 01:43:05.093807 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 217/433] 10.222.0.27 () {56 vars in 1099 bytes} [Tue Aug 30 01:43:04 2022] GET /identity/v3/roles/member => generated 84 bytes in 178 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 01:43:05.107208 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9043ef2-a332-4795-ba2c-5badcae74698 None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:05.170617 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c35f01eb-6de0-423f-8e95-0db8a44aedbc None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:05.218154 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38bd2670-1037-44c9-81a8-550969516ef3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:05.218797 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38bd2670-1037-44c9-81a8-550969516ef3 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:05.219072 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38bd2670-1037-44c9-81a8-550969516ef3 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:05.219356 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38bd2670-1037-44c9-81a8-550969516ef3 None admin] PATH_INFO: `/v3/projects/90818f998d5f4520971eacf6ccc830ce` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:05.221167 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38bd2670-1037-44c9-81a8-550969516ef3 None admin] RBAC: Authorizing `identity:get_project(project_id=90818f998d5f4520971eacf6ccc830ce)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:05.223551 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38bd2670-1037-44c9-81a8-550969516ef3 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:05.227749 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 217/434] 10.222.0.27 () {56 vars in 1186 bytes} [Tue Aug 30 01:43:05 2022] GET /identity/v3/projects/90818f998d5f4520971eacf6ccc830ce => generated 298 bytes in 199 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 30 01:43:05.228414 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38bd2670-1037-44c9-81a8-550969516ef3 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:05.258085 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9043ef2-a332-4795-ba2c-5badcae74698 None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:05.265113 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:43:05.274561 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:43:05.296537 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f709b4c-319b-47cf-ba90-4289200a6aa5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:05.297151 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f709b4c-319b-47cf-ba90-4289200a6aa5 None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:05.297432 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f709b4c-319b-47cf-ba90-4289200a6aa5 None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:05.297733 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f709b4c-319b-47cf-ba90-4289200a6aa5 None admin] PATH_INFO: `/v3/roles` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:05.298398 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f709b4c-319b-47cf-ba90-4289200a6aa5 None admin] RBAC: Adding query filter params (name=member) {{(pid=81578) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:43:05.298700 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f709b4c-319b-47cf-ba90-4289200a6aa5 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:05.300065 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f709b4c-319b-47cf-ba90-4289200a6aa5 None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:05.313598 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 218/435] 10.222.0.27 () {56 vars in 1101 bytes} [Tue Aug 30 01:43:05 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 202 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:43:05.313598 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f709b4c-319b-47cf-ba90-4289200a6aa5 None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:05.342005 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38bd2670-1037-44c9-81a8-550969516ef3 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:05.381591 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-295c3eba-6f0d-4cbc-9128-59535628f3f9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:05.382786 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-295c3eba-6f0d-4cbc-9128-59535628f3f9 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:05.383218 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-295c3eba-6f0d-4cbc-9128-59535628f3f9 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:05.383701 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-295c3eba-6f0d-4cbc-9128-59535628f3f9 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:05.395717 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-295c3eba-6f0d-4cbc-9128-59535628f3f9 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 30 01:43:05.401685 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application [None req-295c3eba-6f0d-4cbc-9128-59535628f3f9 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Aug 30 01:43:05.401685 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 01:43:05.401685 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 01:43:05.401685 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 01:43:05.401685 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 01:43:05.401685 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 01:43:05.401685 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 01:43:05.401685 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 01:43:05.401685 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 01:43:05.401685 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 01:43:05.401685 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 01:43:05.401685 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 01:43:05.401685 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 30 01:43:05.401685 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 30 01:43:05.401685 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 30 01:43:05.401685 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 30 01:43:05.402745 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 30 01:43:05.402745 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 30 01:43:05.402745 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 30 01:43:05.402745 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 30 01:43:05.402745 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 01:43:05.402745 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 01:43:05.402745 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 01:43:05.402745 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 01:43:05.402745 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 01:43:05.402745 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 01:43:05.402745 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 01:43:05.402745 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application with Lock( Aug 30 01:43:05.402745 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 01:43:05.402745 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._enter() Aug 30 01:43:05.402745 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 01:43:05.402745 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 01:43:05.403564 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 01:43:05.403564 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.creator() Aug 30 01:43:05.403564 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 01:43:05.403564 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application created_value = creator( Aug 30 01:43:05.403564 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 30 01:43:05.403564 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 30 01:43:05.403564 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 30 01:43:05.403564 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 30 01:43:05.403564 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 30 01:43:05.403564 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 30 01:43:05.403564 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Aug 30 01:43:05.403564 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Aug 30 01:43:05.403564 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 218/436] 10.222.0.27 () {56 vars in 1114 bytes} [Tue Aug 30 01:43:05 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 175 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 01:43:05.404255 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-295c3eba-6f0d-4cbc-9128-59535628f3f9 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:05.427851 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f709b4c-319b-47cf-ba90-4289200a6aa5 None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:05.462803 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c35372fd-5a31-4564-a644-ad46d8ba29bf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:05.463435 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c35372fd-5a31-4564-a644-ad46d8ba29bf None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:05.463715 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c35372fd-5a31-4564-a644-ad46d8ba29bf None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:05.463989 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c35372fd-5a31-4564-a644-ad46d8ba29bf None admin] PATH_INFO: `/v3/groups/40df2032c69847bbaab1d9363f127d52` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:05.465119 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c35372fd-5a31-4564-a644-ad46d8ba29bf None admin] RBAC: Authorizing `identity:get_group(group_id=40df2032c69847bbaab1d9363f127d52)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:05.466815 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c35372fd-5a31-4564-a644-ad46d8ba29bf None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:05.468191 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 219/437] 10.222.0.27 () {56 vars in 1180 bytes} [Tue Aug 30 01:43:05 2022] GET /identity/v3/groups/40df2032c69847bbaab1d9363f127d52 => generated 226 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:43:05.468953 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c35372fd-5a31-4564-a644-ad46d8ba29bf None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:05.510100 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:43:05.512782 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:43:05.519222 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-295c3eba-6f0d-4cbc-9128-59535628f3f9 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:05.564465 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c2ad396-c5c8-4a60-b1f3-5baea6ffa69e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:05.565076 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c2ad396-c5c8-4a60-b1f3-5baea6ffa69e None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:05.565350 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c2ad396-c5c8-4a60-b1f3-5baea6ffa69e None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:05.565646 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c2ad396-c5c8-4a60-b1f3-5baea6ffa69e None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:05.568646 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c35372fd-5a31-4564-a644-ad46d8ba29bf None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:05.603272 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f272dda6-b226-4832-8a99-720f06274e07 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:05.603869 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f272dda6-b226-4832-8a99-720f06274e07 None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:05.604145 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f272dda6-b226-4832-8a99-720f06274e07 None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:05.604411 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f272dda6-b226-4832-8a99-720f06274e07 None admin] PATH_INFO: `/v3/roles` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:05.605067 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f272dda6-b226-4832-8a99-720f06274e07 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81578) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:43:05.605378 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f272dda6-b226-4832-8a99-720f06274e07 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:05.606744 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f272dda6-b226-4832-8a99-720f06274e07 None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:05.614480 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 220/438] 10.222.0.27 () {56 vars in 1111 bytes} [Tue Aug 30 01:43:05 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:43:05.615608 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e306ffdb-4d7f-4073-bcfb-8ea855fc7b66 None None] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:05.615885 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e306ffdb-4d7f-4073-bcfb-8ea855fc7b66 None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:05.616153 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e306ffdb-4d7f-4073-bcfb-8ea855fc7b66 None None] PATH_INFO: `/` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:05.618375 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 221/439] 10.222.0.27 () {54 vars in 927 bytes} [Tue Aug 30 01:43:05 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 01:43:05.620050 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e306ffdb-4d7f-4073-bcfb-8ea855fc7b66 None None] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:05.639893 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c2ad396-c5c8-4a60-b1f3-5baea6ffa69e None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:05.641354 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c2ad396-c5c8-4a60-b1f3-5baea6ffa69e None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:05.726903 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e306ffdb-4d7f-4073-bcfb-8ea855fc7b66 None None] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:05.729195 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:43:05.733039 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:43:05.742929 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 219/440] 10.222.0.27 () {58 vars in 1282 bytes} [Tue Aug 30 01:43:05 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 339 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 30 01:43:05.743743 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c2ad396-c5c8-4a60-b1f3-5baea6ffa69e None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:05.760117 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06c82a93-fef4-492a-af6d-f186e8d69b22 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:05.760777 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06c82a93-fef4-492a-af6d-f186e8d69b22 None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:05.761054 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06c82a93-fef4-492a-af6d-f186e8d69b22 None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:05.761744 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06c82a93-fef4-492a-af6d-f186e8d69b22 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:05.762953 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06c82a93-fef4-492a-af6d-f186e8d69b22 None admin] RBAC: Adding query filter params (group.id=40df2032c69847bbaab1d9363f127d52, role.id=1fb81afcb9f245f3b989add94598a063, scope.project.id=90818f998d5f4520971eacf6ccc830ce) {{(pid=81578) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:43:05.763599 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06c82a93-fef4-492a-af6d-f186e8d69b22 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:05.765210 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06c82a93-fef4-492a-af6d-f186e8d69b22 None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:05.770930 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 222/441] 10.222.0.27 () {56 vars in 1376 bytes} [Tue Aug 30 01:43:05 2022] GET /identity/v3/role_assignments?group.id=40df2032c69847bbaab1d9363f127d52&scope.project.id=90818f998d5f4520971eacf6ccc830ce&role.id=1fb81afcb9f245f3b989add94598a063 => generated 611 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:43:05.781647 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36cf425a-8976-44bd-b6c4-078137b4de1e None None] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:05.781647 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36cf425a-8976-44bd-b6c4-078137b4de1e None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:05.781647 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36cf425a-8976-44bd-b6c4-078137b4de1e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:05.802698 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.auth.core [None req-36cf425a-8976-44bd-b6c4-078137b4de1e None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81578) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 01:43:05.803937 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-36cf425a-8976-44bd-b6c4-078137b4de1e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:43:05.853848 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c2ad396-c5c8-4a60-b1f3-5baea6ffa69e None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:05.879885 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 223/442] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Aug 30 01:43:05 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 30 01:43:05.880875 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36cf425a-8976-44bd-b6c4-078137b4de1e None None] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:05.900112 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f029b1e1-523c-4eb2-a59d-f8ba355bd26c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:05.901107 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f029b1e1-523c-4eb2-a59d-f8ba355bd26c None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:05.901612 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f029b1e1-523c-4eb2-a59d-f8ba355bd26c None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:05.902082 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f029b1e1-523c-4eb2-a59d-f8ba355bd26c None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:05.977537 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:43:05.988345 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f029b1e1-523c-4eb2-a59d-f8ba355bd26c None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:05.991441 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f029b1e1-523c-4eb2-a59d-f8ba355bd26c None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:06.035965 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:43:06.040720 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36cf425a-8976-44bd-b6c4-078137b4de1e None None] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:06.084883 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c53870d-7a2a-4462-9203-5bf9305b15c8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:06.086124 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c53870d-7a2a-4462-9203-5bf9305b15c8 None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:06.086408 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c53870d-7a2a-4462-9203-5bf9305b15c8 None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:06.086679 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c53870d-7a2a-4462-9203-5bf9305b15c8 None admin] PATH_INFO: `/v3/projects/67913d5682a443069a5eed96f179751f` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:06.087883 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c53870d-7a2a-4462-9203-5bf9305b15c8 None admin] RBAC: Authorizing `identity:get_project(project_id=67913d5682a443069a5eed96f179751f)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:06.090589 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c53870d-7a2a-4462-9203-5bf9305b15c8 None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:06.092818 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 224/443] 10.222.0.27 () {56 vars in 1186 bytes} [Tue Aug 30 01:43:05 2022] GET /identity/v3/projects/67913d5682a443069a5eed96f179751f => generated 302 bytes in 212 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:43:06.098523 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c53870d-7a2a-4462-9203-5bf9305b15c8 None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:06.127752 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 220/444] 10.222.0.27 () {58 vars in 1282 bytes} [Tue Aug 30 01:43:05 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 384 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 30 01:43:06.134638 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f029b1e1-523c-4eb2-a59d-f8ba355bd26c None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:06.224726 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c53870d-7a2a-4462-9203-5bf9305b15c8 None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:06.250230 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:43:06.263433 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f029b1e1-523c-4eb2-a59d-f8ba355bd26c None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:06.265391 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87ecba2f-39a0-40d7-a2d6-88b50f0c7d00 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:06.266040 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87ecba2f-39a0-40d7-a2d6-88b50f0c7d00 None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:06.266318 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87ecba2f-39a0-40d7-a2d6-88b50f0c7d00 None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:06.266587 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87ecba2f-39a0-40d7-a2d6-88b50f0c7d00 None admin] PATH_INFO: `/v3/groups/40df2032c69847bbaab1d9363f127d52` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:06.267693 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87ecba2f-39a0-40d7-a2d6-88b50f0c7d00 None admin] RBAC: Authorizing `identity:get_group(group_id=40df2032c69847bbaab1d9363f127d52)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:06.269115 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87ecba2f-39a0-40d7-a2d6-88b50f0c7d00 None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:06.270709 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 225/445] 10.222.0.27 () {56 vars in 1180 bytes} [Tue Aug 30 01:43:06 2022] GET /identity/v3/groups/40df2032c69847bbaab1d9363f127d52 => generated 226 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 30 01:43:06.271461 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87ecba2f-39a0-40d7-a2d6-88b50f0c7d00 None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:06.306797 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3088253b-fd98-4a10-b6d9-dac11ec1cb82 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:06.307753 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3088253b-fd98-4a10-b6d9-dac11ec1cb82 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:06.308204 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3088253b-fd98-4a10-b6d9-dac11ec1cb82 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:06.308648 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3088253b-fd98-4a10-b6d9-dac11ec1cb82 None admin] PATH_INFO: `/v3/projects/67913d5682a443069a5eed96f179751f` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:06.309148 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:43:06.310093 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3088253b-fd98-4a10-b6d9-dac11ec1cb82 None admin] RBAC: Authorizing `identity:get_project(project_id=67913d5682a443069a5eed96f179751f)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:06.311741 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3088253b-fd98-4a10-b6d9-dac11ec1cb82 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:06.313356 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 221/446] 10.222.0.27 () {56 vars in 1186 bytes} [Tue Aug 30 01:43:06 2022] GET /identity/v3/projects/67913d5682a443069a5eed96f179751f => generated 302 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 30 01:43:06.317621 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3088253b-fd98-4a10-b6d9-dac11ec1cb82 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:06.387097 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87ecba2f-39a0-40d7-a2d6-88b50f0c7d00 None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:06.420523 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-846271e7-54d7-4b5f-8eba-296c39d436c4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:06.421152 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-846271e7-54d7-4b5f-8eba-296c39d436c4 None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:06.421432 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-846271e7-54d7-4b5f-8eba-296c39d436c4 None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:06.421730 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-846271e7-54d7-4b5f-8eba-296c39d436c4 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:06.429780 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3088253b-fd98-4a10-b6d9-dac11ec1cb82 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:06.431963 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-846271e7-54d7-4b5f-8eba-296c39d436c4 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 30 01:43:06.436784 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application [None req-846271e7-54d7-4b5f-8eba-296c39d436c4 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 30 01:43:06.436784 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 01:43:06.436784 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 01:43:06.436784 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 01:43:06.436784 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 01:43:06.436784 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 01:43:06.436784 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 01:43:06.436784 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 01:43:06.436784 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 01:43:06.436784 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 01:43:06.436784 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 01:43:06.436784 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 01:43:06.436784 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 30 01:43:06.436784 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 30 01:43:06.436784 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 30 01:43:06.436784 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 30 01:43:06.437840 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 30 01:43:06.437840 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 30 01:43:06.437840 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 30 01:43:06.437840 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 30 01:43:06.437840 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 01:43:06.437840 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 01:43:06.437840 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 01:43:06.437840 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 01:43:06.437840 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 01:43:06.437840 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 01:43:06.437840 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 01:43:06.437840 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application with Lock( Aug 30 01:43:06.437840 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 01:43:06.437840 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._enter() Aug 30 01:43:06.437840 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 01:43:06.437840 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 01:43:06.438656 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 01:43:06.438656 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.creator() Aug 30 01:43:06.438656 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 01:43:06.438656 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application created_value = creator( Aug 30 01:43:06.438656 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 30 01:43:06.438656 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 30 01:43:06.438656 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 30 01:43:06.438656 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 30 01:43:06.438656 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 30 01:43:06.438656 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 30 01:43:06.438656 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 30 01:43:06.438656 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application Aug 30 01:43:06.438656 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 226/447] 10.222.0.27 () {56 vars in 1096 bytes} [Tue Aug 30 01:43:06 2022] GET /identity/v3/roles/admin => generated 83 bytes in 167 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 01:43:06.438656 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-846271e7-54d7-4b5f-8eba-296c39d436c4 None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:06.465266 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-286ae4cb-d8b8-4f46-a4f4-5c6e9b309509 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:06.465893 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-286ae4cb-d8b8-4f46-a4f4-5c6e9b309509 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:06.466172 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-286ae4cb-d8b8-4f46-a4f4-5c6e9b309509 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:06.466442 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-286ae4cb-d8b8-4f46-a4f4-5c6e9b309509 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:06.467168 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-286ae4cb-d8b8-4f46-a4f4-5c6e9b309509 None admin] RBAC: Adding query filter params (group.id=40df2032c69847bbaab1d9363f127d52, role.id=1fb81afcb9f245f3b989add94598a063, scope.project.id=67913d5682a443069a5eed96f179751f) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:43:06.467479 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-286ae4cb-d8b8-4f46-a4f4-5c6e9b309509 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:06.468891 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-286ae4cb-d8b8-4f46-a4f4-5c6e9b309509 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:06.477717 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 222/448] 10.222.0.27 () {56 vars in 1376 bytes} [Tue Aug 30 01:43:06 2022] GET /identity/v3/role_assignments?group.id=40df2032c69847bbaab1d9363f127d52&scope.project.id=67913d5682a443069a5eed96f179751f&role.id=1fb81afcb9f245f3b989add94598a063 => generated 611 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:43:06.477717 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-286ae4cb-d8b8-4f46-a4f4-5c6e9b309509 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:06.509227 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:43:06.550107 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:43:06.558557 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-846271e7-54d7-4b5f-8eba-296c39d436c4 None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:06.596774 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f52b02bd-7d06-4115-b6d9-6a6f817ceb2f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:06.597407 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f52b02bd-7d06-4115-b6d9-6a6f817ceb2f None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:06.597714 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f52b02bd-7d06-4115-b6d9-6a6f817ceb2f None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:06.597984 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f52b02bd-7d06-4115-b6d9-6a6f817ceb2f None admin] PATH_INFO: `/v3/groups/40df2032c69847bbaab1d9363f127d52` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:06.598254 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-286ae4cb-d8b8-4f46-a4f4-5c6e9b309509 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:06.599137 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f52b02bd-7d06-4115-b6d9-6a6f817ceb2f None admin] RBAC: Authorizing `identity:get_group(group_id=40df2032c69847bbaab1d9363f127d52)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:06.600587 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f52b02bd-7d06-4115-b6d9-6a6f817ceb2f None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:06.603035 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 227/449] 10.222.0.27 () {56 vars in 1180 bytes} [Tue Aug 30 01:43:06 2022] GET /identity/v3/groups/40df2032c69847bbaab1d9363f127d52 => generated 226 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:43:06.608876 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f52b02bd-7d06-4115-b6d9-6a6f817ceb2f None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:06.640142 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-757161b7-5828-480c-8ccf-0ad0f35f0aa0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:06.640741 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-757161b7-5828-480c-8ccf-0ad0f35f0aa0 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:06.641021 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-757161b7-5828-480c-8ccf-0ad0f35f0aa0 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:06.641292 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-757161b7-5828-480c-8ccf-0ad0f35f0aa0 None admin] PATH_INFO: `/v3/roles` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:06.642040 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-757161b7-5828-480c-8ccf-0ad0f35f0aa0 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:43:06.642299 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-757161b7-5828-480c-8ccf-0ad0f35f0aa0 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:06.643718 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-757161b7-5828-480c-8ccf-0ad0f35f0aa0 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:06.654720 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 223/450] 10.222.0.27 () {56 vars in 1099 bytes} [Tue Aug 30 01:43:06 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 180 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:43:06.662256 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-757161b7-5828-480c-8ccf-0ad0f35f0aa0 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:06.743623 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f52b02bd-7d06-4115-b6d9-6a6f817ceb2f None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:06.765538 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:43:06.785790 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-757161b7-5828-480c-8ccf-0ad0f35f0aa0 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:06.787591 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d05d3121-ed0f-4c16-aaae-d704f3240c27 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:06.788414 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d05d3121-ed0f-4c16-aaae-d704f3240c27 None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:06.789386 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d05d3121-ed0f-4c16-aaae-d704f3240c27 None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:06.789753 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d05d3121-ed0f-4c16-aaae-d704f3240c27 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:06.790576 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d05d3121-ed0f-4c16-aaae-d704f3240c27 None admin] RBAC: Adding query filter params (group.id=40df2032c69847bbaab1d9363f127d52, role.id=4621068972ba4f79bad104f956db04aa, scope.project.id=67913d5682a443069a5eed96f179751f) {{(pid=81578) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:43:06.790884 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d05d3121-ed0f-4c16-aaae-d704f3240c27 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:06.792310 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d05d3121-ed0f-4c16-aaae-d704f3240c27 None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:06.802581 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 228/451] 10.222.0.27 () {56 vars in 1376 bytes} [Tue Aug 30 01:43:06 2022] GET /identity/v3/role_assignments?group.id=40df2032c69847bbaab1d9363f127d52&scope.project.id=67913d5682a443069a5eed96f179751f&role.id=4621068972ba4f79bad104f956db04aa => generated 611 bytes in 194 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 30 01:43:06.824390 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e1a2fea-c301-4ae7-9792-b344653321ca None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:06.825014 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e1a2fea-c301-4ae7-9792-b344653321ca None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:06.825293 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e1a2fea-c301-4ae7-9792-b344653321ca None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:06.825595 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e1a2fea-c301-4ae7-9792-b344653321ca None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:06.903332 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e1a2fea-c301-4ae7-9792-b344653321ca None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:06.904814 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e1a2fea-c301-4ae7-9792-b344653321ca None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:06.989849 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:43:07.005915 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 224/452] 10.222.0.27 () {58 vars in 1282 bytes} [Tue Aug 30 01:43:06 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 344 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Aug 30 01:43:07.012043 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d05d3121-ed0f-4c16-aaae-d704f3240c27 None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:07.041527 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:43:07.117443 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d05d3121-ed0f-4c16-aaae-d704f3240c27 None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:07.152139 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3138751-9f64-4721-ac08-0a60f2856bae None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:07.152762 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3138751-9f64-4721-ac08-0a60f2856bae None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:07.153040 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3138751-9f64-4721-ac08-0a60f2856bae None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:07.153309 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3138751-9f64-4721-ac08-0a60f2856bae None admin] PATH_INFO: `/v3/projects/70bc9f42895844308fb4b9192314f80b` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:07.154435 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3138751-9f64-4721-ac08-0a60f2856bae None admin] RBAC: Authorizing `identity:get_project(project_id=70bc9f42895844308fb4b9192314f80b)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:07.155849 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3138751-9f64-4721-ac08-0a60f2856bae None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:07.157506 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 229/453] 10.222.0.27 () {56 vars in 1186 bytes} [Tue Aug 30 01:43:07 2022] GET /identity/v3/projects/70bc9f42895844308fb4b9192314f80b => generated 344 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:43:07.163035 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e1a2fea-c301-4ae7-9792-b344653321ca None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:07.258474 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e1a2fea-c301-4ae7-9792-b344653321ca None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:07.291043 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83f3540c-9639-4b4a-a99a-acaa2dbbfb13 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:07.291700 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83f3540c-9639-4b4a-a99a-acaa2dbbfb13 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:07.291929 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83f3540c-9639-4b4a-a99a-acaa2dbbfb13 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:07.292196 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83f3540c-9639-4b4a-a99a-acaa2dbbfb13 None admin] PATH_INFO: `/v3/groups/e130c65bd71a4dc983e47fda13b8d556` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:07.293544 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83f3540c-9639-4b4a-a99a-acaa2dbbfb13 None admin] RBAC: Authorizing `identity:get_group(group_id=e130c65bd71a4dc983e47fda13b8d556)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:07.294988 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83f3540c-9639-4b4a-a99a-acaa2dbbfb13 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:07.296517 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 225/454] 10.222.0.27 () {56 vars in 1180 bytes} [Tue Aug 30 01:43:07 2022] GET /identity/v3/groups/e130c65bd71a4dc983e47fda13b8d556 => generated 229 bytes in 134 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:43:07.302292 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3138751-9f64-4721-ac08-0a60f2856bae None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:07.392999 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:43:07.399891 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3138751-9f64-4721-ac08-0a60f2856bae None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:07.431941 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a24bebd-b8b4-4c0b-b264-340d63cc5b30 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:07.432532 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a24bebd-b8b4-4c0b-b264-340d63cc5b30 None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:07.432808 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a24bebd-b8b4-4c0b-b264-340d63cc5b30 None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:07.433077 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a24bebd-b8b4-4c0b-b264-340d63cc5b30 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:07.433826 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a24bebd-b8b4-4c0b-b264-340d63cc5b30 None admin] RBAC: Adding query filter params (group.id=e130c65bd71a4dc983e47fda13b8d556, role.id=ead5fac86a0747ea8823a82efcea8ed7, scope.project.id=70bc9f42895844308fb4b9192314f80b) {{(pid=81578) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:43:07.434132 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a24bebd-b8b4-4c0b-b264-340d63cc5b30 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:07.435565 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a24bebd-b8b4-4c0b-b264-340d63cc5b30 None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:07.441105 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 230/455] 10.222.0.27 () {56 vars in 1376 bytes} [Tue Aug 30 01:43:07 2022] GET /identity/v3/role_assignments?group.id=e130c65bd71a4dc983e47fda13b8d556&scope.project.id=70bc9f42895844308fb4b9192314f80b&role.id=ead5fac86a0747ea8823a82efcea8ed7 => generated 611 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:43:10.010685 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-691e583a-0928-4e40-bf5a-db1fe2f3cbf8 None None] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:10.011996 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-691e583a-0928-4e40-bf5a-db1fe2f3cbf8 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:10.012664 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-691e583a-0928-4e40-bf5a-db1fe2f3cbf8 None None] PATH_INFO: `/` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:10.015333 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 226/456] 10.222.0.27 () {54 vars in 927 bytes} [Tue Aug 30 01:43:10 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 01:43:10.020890 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a64cd7e2-c753-453b-8787-c6f05d08b1ff None None] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:10.022183 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a64cd7e2-c753-453b-8787-c6f05d08b1ff None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:10.023452 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a64cd7e2-c753-453b-8787-c6f05d08b1ff None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:10.039943 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a4b48c7-70f6-4fa5-836c-f60fb423c959 None None] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:10.040558 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a4b48c7-70f6-4fa5-836c-f60fb423c959 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:10.041024 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a4b48c7-70f6-4fa5-836c-f60fb423c959 None None] PATH_INFO: `/` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:10.042587 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 227/457] 10.222.0.27 () {54 vars in 927 bytes} [Tue Aug 30 01:43:10 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 01:43:10.048684 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b483bfee-4144-47ac-98fe-aa140e9ea2ae None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:10.049280 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b483bfee-4144-47ac-98fe-aa140e9ea2ae None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:10.049777 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b483bfee-4144-47ac-98fe-aa140e9ea2ae None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:10.061506 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.auth.core [None req-a64cd7e2-c753-453b-8787-c6f05d08b1ff None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81578) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 01:43:10.063921 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-a64cd7e2-c753-453b-8787-c6f05d08b1ff None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:43:10.082893 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-b483bfee-4144-47ac-98fe-aa140e9ea2ae None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 01:43:10.084412 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-b483bfee-4144-47ac-98fe-aa140e9ea2ae None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:43:10.146037 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:43:10.153325 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 231/458] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Aug 30 01:43:10 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 134 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 30 01:43:10.154975 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0dc4d45-fe46-4778-84a5-7c052f3c40b3 None None] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:10.265509 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0dc4d45-fe46-4778-84a5-7c052f3c40b3 None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:10.265509 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0dc4d45-fe46-4778-84a5-7c052f3c40b3 None None] PATH_INFO: `/` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:10.265509 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 232/459] 10.222.0.27 () {54 vars in 927 bytes} [Tue Aug 30 01:43:10 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 01:43:10.265509 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65c4ea68-49da-4c34-af52-6e867815b50a None None] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:10.265509 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65c4ea68-49da-4c34-af52-6e867815b50a None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:10.265509 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65c4ea68-49da-4c34-af52-6e867815b50a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:10.265965 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 228/460] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Aug 30 01:43:10 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 128 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 30 01:43:10.266039 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.auth.core [None req-65c4ea68-49da-4c34-af52-6e867815b50a None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81578) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 01:43:10.266039 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-65c4ea68-49da-4c34-af52-6e867815b50a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:43:10.284240 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 233/461] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Aug 30 01:43:10 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 122 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Aug 30 01:43:10.499530 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b483bfee-4144-47ac-98fe-aa140e9ea2ae None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:10.515345 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65c4ea68-49da-4c34-af52-6e867815b50a None None] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:10.541661 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:43:10.634001 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b483bfee-4144-47ac-98fe-aa140e9ea2ae None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:10.665664 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65c4ea68-49da-4c34-af52-6e867815b50a None None] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:10.670868 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee6298de-05af-40af-87ee-380354d3a29d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:10.671747 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee6298de-05af-40af-87ee-380354d3a29d None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:10.672205 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee6298de-05af-40af-87ee-380354d3a29d None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:10.672646 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee6298de-05af-40af-87ee-380354d3a29d None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:10.684015 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee6298de-05af-40af-87ee-380354d3a29d None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:10.685737 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee6298de-05af-40af-87ee-380354d3a29d None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:10.711409 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application [None req-ee6298de-05af-40af-87ee-380354d3a29d None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 01:43:10.711409 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 01:43:10.711409 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 01:43:10.711409 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 01:43:10.711409 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 01:43:10.711409 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 01:43:10.711409 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 01:43:10.711409 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 01:43:10.711409 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 01:43:10.711409 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 01:43:10.711409 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 01:43:10.711409 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 01:43:10.711409 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 30 01:43:10.711409 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 30 01:43:10.711409 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 30 01:43:10.711409 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 30 01:43:10.712385 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 01:43:10.712385 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 01:43:10.712385 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 01:43:10.712385 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 01:43:10.712385 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 01:43:10.712385 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 01:43:10.712385 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 01:43:10.712385 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application with Lock( Aug 30 01:43:10.712385 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 01:43:10.712385 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._enter() Aug 30 01:43:10.712385 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 01:43:10.712385 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 01:43:10.712385 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 01:43:10.712385 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.creator() Aug 30 01:43:10.712385 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 01:43:10.712385 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application created_value = creator( Aug 30 01:43:10.712385 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 30 01:43:10.713238 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 30 01:43:10.713238 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 01:43:10.713238 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Aug 30 01:43:10.715942 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 229/462] 10.222.0.27 () {56 vars in 1108 bytes} [Tue Aug 30 01:43:10 2022] GET /identity/v3/domains/Default => generated 87 bytes in 217 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 01:43:10.719581 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee6298de-05af-40af-87ee-380354d3a29d None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:10.719739 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9736c91b-ed79-4ad1-8285-f5818d6ac0fb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:10.719739 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9736c91b-ed79-4ad1-8285-f5818d6ac0fb None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:10.719739 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9736c91b-ed79-4ad1-8285-f5818d6ac0fb None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:10.719739 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9736c91b-ed79-4ad1-8285-f5818d6ac0fb None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:10.733824 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9736c91b-ed79-4ad1-8285-f5818d6ac0fb None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:10.735563 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9736c91b-ed79-4ad1-8285-f5818d6ac0fb None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:10.753066 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application [None req-9736c91b-ed79-4ad1-8285-f5818d6ac0fb None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 01:43:10.753066 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 01:43:10.753066 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 01:43:10.753066 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 01:43:10.753066 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 01:43:10.753066 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 01:43:10.753066 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 01:43:10.753066 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 01:43:10.753066 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 01:43:10.753066 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 01:43:10.753066 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 01:43:10.753066 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 01:43:10.753066 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 30 01:43:10.753066 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 30 01:43:10.753066 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 30 01:43:10.753066 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 30 01:43:10.754053 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 01:43:10.754053 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 01:43:10.754053 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 01:43:10.754053 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 01:43:10.754053 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 01:43:10.754053 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 01:43:10.754053 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 01:43:10.754053 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application with Lock( Aug 30 01:43:10.754053 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 01:43:10.754053 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._enter() Aug 30 01:43:10.754053 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 01:43:10.754053 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 01:43:10.754053 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 01:43:10.754053 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.creator() Aug 30 01:43:10.754053 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 01:43:10.754053 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application created_value = creator( Aug 30 01:43:10.754053 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 30 01:43:10.754915 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 30 01:43:10.754915 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 01:43:10.754915 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application Aug 30 01:43:10.757709 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 234/463] 10.222.0.27 () {56 vars in 1108 bytes} [Tue Aug 30 01:43:10 2022] GET /identity/v3/domains/Default => generated 87 bytes in 242 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 30 01:43:10.758901 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9736c91b-ed79-4ad1-8285-f5818d6ac0fb None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:10.768267 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:43:10.845546 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:43:10.845829 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee6298de-05af-40af-87ee-380354d3a29d None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:10.880165 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9736c91b-ed79-4ad1-8285-f5818d6ac0fb None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:10.884546 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f43d5682-f0a6-4c75-964d-acd1c39f7692 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:10.885177 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f43d5682-f0a6-4c75-964d-acd1c39f7692 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:10.885484 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f43d5682-f0a6-4c75-964d-acd1c39f7692 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:10.885751 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f43d5682-f0a6-4c75-964d-acd1c39f7692 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:10.896009 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f43d5682-f0a6-4c75-964d-acd1c39f7692 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:10.897490 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f43d5682-f0a6-4c75-964d-acd1c39f7692 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:10.912644 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application [None req-f43d5682-f0a6-4c75-964d-acd1c39f7692 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 01:43:10.912644 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 01:43:10.912644 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 01:43:10.912644 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 01:43:10.912644 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 01:43:10.912644 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 01:43:10.912644 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 01:43:10.912644 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 01:43:10.912644 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 01:43:10.912644 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 01:43:10.912644 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 01:43:10.912644 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 01:43:10.912644 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 30 01:43:10.912644 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 30 01:43:10.912644 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 30 01:43:10.912644 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 30 01:43:10.913937 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 01:43:10.913937 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 01:43:10.913937 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 01:43:10.913937 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 01:43:10.913937 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 01:43:10.913937 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 01:43:10.913937 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 01:43:10.913937 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application with Lock( Aug 30 01:43:10.913937 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 01:43:10.913937 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._enter() Aug 30 01:43:10.913937 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 01:43:10.913937 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 01:43:10.913937 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 01:43:10.913937 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.creator() Aug 30 01:43:10.913937 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 01:43:10.913937 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application created_value = creator( Aug 30 01:43:10.913937 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 30 01:43:10.914915 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 30 01:43:10.914915 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 01:43:10.914915 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Aug 30 01:43:10.914915 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 230/464] 10.222.0.27 () {56 vars in 1108 bytes} [Tue Aug 30 01:43:10 2022] GET /identity/v3/domains/Default => generated 87 bytes in 198 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 01:43:10.914915 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f43d5682-f0a6-4c75-964d-acd1c39f7692 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:10.921326 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57743847-3aff-4a39-9271-0de90e1f9868 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:10.921973 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57743847-3aff-4a39-9271-0de90e1f9868 None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:10.922254 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57743847-3aff-4a39-9271-0de90e1f9868 None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:10.922526 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57743847-3aff-4a39-9271-0de90e1f9868 None admin] PATH_INFO: `/v3/domains` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:10.923179 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57743847-3aff-4a39-9271-0de90e1f9868 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81578) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:43:10.923506 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57743847-3aff-4a39-9271-0de90e1f9868 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:10.924872 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57743847-3aff-4a39-9271-0de90e1f9868 None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:10.937965 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 235/465] 10.222.0.27 () {56 vars in 1109 bytes} [Tue Aug 30 01:43:10 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:43:10.939262 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2905373-3481-40e5-a15e-c38940a7221f None None] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:10.939550 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2905373-3481-40e5-a15e-c38940a7221f None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:10.939821 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2905373-3481-40e5-a15e-c38940a7221f None None] PATH_INFO: `/` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:10.941187 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 236/466] 10.222.0.27 () {54 vars in 927 bytes} [Tue Aug 30 01:43:10 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 01:43:10.945519 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2905373-3481-40e5-a15e-c38940a7221f None None] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:11.020069 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:43:11.037793 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f43d5682-f0a6-4c75-964d-acd1c39f7692 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:11.049150 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2905373-3481-40e5-a15e-c38940a7221f None None] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:11.076226 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-131bad55-c026-4694-be53-6f65b0a719a0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:11.076891 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-131bad55-c026-4694-be53-6f65b0a719a0 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:11.077168 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-131bad55-c026-4694-be53-6f65b0a719a0 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:11.077439 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-131bad55-c026-4694-be53-6f65b0a719a0 None admin] PATH_INFO: `/v3/domains` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:11.078104 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:43:11.078916 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-131bad55-c026-4694-be53-6f65b0a719a0 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:43:11.079202 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-131bad55-c026-4694-be53-6f65b0a719a0 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:11.080628 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-131bad55-c026-4694-be53-6f65b0a719a0 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:11.084031 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cd7ec6a-e3a1-4328-832c-4e6002af2dfa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:11.084628 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cd7ec6a-e3a1-4328-832c-4e6002af2dfa None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:11.084906 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cd7ec6a-e3a1-4328-832c-4e6002af2dfa None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:11.085778 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cd7ec6a-e3a1-4328-832c-4e6002af2dfa None admin] PATH_INFO: `/v3/domains` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:11.094529 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2cd7ec6a-e3a1-4328-832c-4e6002af2dfa None admin] RBAC: Adding query filter params (name=Default) {{(pid=81578) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:43:11.094529 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2cd7ec6a-e3a1-4328-832c-4e6002af2dfa None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:11.094529 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2cd7ec6a-e3a1-4328-832c-4e6002af2dfa None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:11.094765 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 231/467] 10.222.0.27 () {56 vars in 1109 bytes} [Tue Aug 30 01:43:10 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:43:11.094765 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-131bad55-c026-4694-be53-6f65b0a719a0 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:11.103910 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 237/468] 10.222.0.27 () {56 vars in 1109 bytes} [Tue Aug 30 01:43:10 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:43:11.103910 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6c09b1c-563a-4df9-90b6-bdb4e40b98ef None None] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:11.103910 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6c09b1c-563a-4df9-90b6-bdb4e40b98ef None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:11.103910 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6c09b1c-563a-4df9-90b6-bdb4e40b98ef None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:11.123513 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.auth.core [None req-f6c09b1c-563a-4df9-90b6-bdb4e40b98ef None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81578) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 01:43:11.124878 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-f6c09b1c-563a-4df9-90b6-bdb4e40b98ef None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:43:11.206134 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 238/469] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Aug 30 01:43:11 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 30 01:43:11.207251 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6c09b1c-563a-4df9-90b6-bdb4e40b98ef None None] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:11.226078 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-131bad55-c026-4694-be53-6f65b0a719a0 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:11.269698 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d696b612-f21e-4a61-85fa-814b847da78f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:11.270329 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d696b612-f21e-4a61-85fa-814b847da78f None admin] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:11.270830 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d696b612-f21e-4a61-85fa-814b847da78f None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:11.270949 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d696b612-f21e-4a61-85fa-814b847da78f None admin] PATH_INFO: `/v3/users` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:11.271653 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d696b612-f21e-4a61-85fa-814b847da78f None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:11.273687 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d696b612-f21e-4a61-85fa-814b847da78f None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:11.331826 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6c09b1c-563a-4df9-90b6-bdb4e40b98ef None None] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:11.353613 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:43:11.368559 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e29de5df-b042-4967-9ce9-5876745c2e7d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:11.369103 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e29de5df-b042-4967-9ce9-5876745c2e7d None admin] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:11.369383 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e29de5df-b042-4967-9ce9-5876745c2e7d None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:11.369693 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e29de5df-b042-4967-9ce9-5876745c2e7d None admin] PATH_INFO: `/v3/users` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:11.370334 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e29de5df-b042-4967-9ce9-5876745c2e7d None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:11.371799 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e29de5df-b042-4967-9ce9-5876745c2e7d None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:11.407840 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 232/470] 10.222.0.27 () {60 vars in 1132 bytes} [Tue Aug 30 01:43:11 2022] POST /identity/v3/users => generated 248 bytes in 314 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 01:43:11.408766 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d696b612-f21e-4a61-85fa-814b847da78f None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:11.411574 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:43:11.471727 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 239/471] 10.222.0.27 () {60 vars in 1132 bytes} [Tue Aug 30 01:43:11 2022] POST /identity/v3/users => generated 246 bytes in 265 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 01:43:11.481353 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75d3d707-5c43-4d5b-a4de-6ae8f43edcb1 None None] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:11.482319 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75d3d707-5c43-4d5b-a4de-6ae8f43edcb1 None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:11.483603 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75d3d707-5c43-4d5b-a4de-6ae8f43edcb1 None None] PATH_INFO: `/` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:11.487637 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 240/472] 10.222.0.27 () {54 vars in 927 bytes} [Tue Aug 30 01:43:11 2022] GET /identity => generated 270 bytes in 9 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 01:43:11.490857 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75d3d707-5c43-4d5b-a4de-6ae8f43edcb1 None None] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:11.583133 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d696b612-f21e-4a61-85fa-814b847da78f None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:11.636555 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b82b218e-0100-42ce-b705-4d87fa7e410d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:11.637488 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b82b218e-0100-42ce-b705-4d87fa7e410d None admin] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:11.637939 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b82b218e-0100-42ce-b705-4d87fa7e410d None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:11.638404 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b82b218e-0100-42ce-b705-4d87fa7e410d None admin] PATH_INFO: `/v3/users` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:11.639442 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b82b218e-0100-42ce-b705-4d87fa7e410d None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:11.640889 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75d3d707-5c43-4d5b-a4de-6ae8f43edcb1 None None] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:11.641283 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b82b218e-0100-42ce-b705-4d87fa7e410d None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:11.679609 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eab1d48b-a36b-4fd5-a032-a4531fb64a02 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:11.680244 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eab1d48b-a36b-4fd5-a032-a4531fb64a02 None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:11.680512 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eab1d48b-a36b-4fd5-a032-a4531fb64a02 None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:11.680785 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eab1d48b-a36b-4fd5-a032-a4531fb64a02 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:11.692148 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eab1d48b-a36b-4fd5-a032-a4531fb64a02 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:11.693656 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eab1d48b-a36b-4fd5-a032-a4531fb64a02 None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:11.715171 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application [None req-eab1d48b-a36b-4fd5-a032-a4531fb64a02 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 01:43:11.715171 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 01:43:11.715171 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 01:43:11.715171 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 01:43:11.715171 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 01:43:11.715171 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 01:43:11.715171 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 01:43:11.715171 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 01:43:11.715171 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 01:43:11.715171 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 01:43:11.715171 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 01:43:11.715171 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 01:43:11.715171 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 30 01:43:11.715171 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 30 01:43:11.715171 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 30 01:43:11.715171 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 30 01:43:11.716113 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 01:43:11.716113 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 01:43:11.716113 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 01:43:11.716113 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 01:43:11.716113 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 01:43:11.716113 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 01:43:11.716113 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 01:43:11.716113 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application with Lock( Aug 30 01:43:11.716113 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 01:43:11.716113 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._enter() Aug 30 01:43:11.716113 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 01:43:11.716113 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 01:43:11.716113 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 01:43:11.716113 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.creator() Aug 30 01:43:11.716113 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 01:43:11.716113 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application created_value = creator( Aug 30 01:43:11.716113 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 30 01:43:11.716991 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 30 01:43:11.716991 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 01:43:11.716991 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application Aug 30 01:43:11.716991 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 241/473] 10.222.0.27 () {56 vars in 1108 bytes} [Tue Aug 30 01:43:11 2022] GET /identity/v3/domains/Default => generated 87 bytes in 226 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 01:43:11.717745 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d4892a8-cd03-4c5a-b778-c184eb49e91a None None] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:11.717841 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d4892a8-cd03-4c5a-b778-c184eb49e91a None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:11.718044 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d4892a8-cd03-4c5a-b778-c184eb49e91a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:11.738810 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.auth.core [None req-1d4892a8-cd03-4c5a-b778-c184eb49e91a None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81578) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 01:43:11.740027 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-1d4892a8-cd03-4c5a-b778-c184eb49e91a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:43:11.765076 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:43:11.809799 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 233/474] 10.222.0.27 () {60 vars in 1132 bytes} [Tue Aug 30 01:43:11 2022] POST /identity/v3/users => generated 248 bytes in 396 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 01:43:11.811583 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b82b218e-0100-42ce-b705-4d87fa7e410d None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:11.831923 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 242/475] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Aug 30 01:43:11 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 115 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 30 01:43:11.893416 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:43:11.940800 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b82b218e-0100-42ce-b705-4d87fa7e410d None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:11.978581 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4aca69d1-85c1-461f-9ca1-36ca9ba02fce None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:11.979273 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4aca69d1-85c1-461f-9ca1-36ca9ba02fce None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:11.979552 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4aca69d1-85c1-461f-9ca1-36ca9ba02fce None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:11.979823 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4aca69d1-85c1-461f-9ca1-36ca9ba02fce None admin] PATH_INFO: `/v3/domains` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:11.980543 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4aca69d1-85c1-461f-9ca1-36ca9ba02fce None admin] RBAC: Adding query filter params (name=Default) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:43:11.980847 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4aca69d1-85c1-461f-9ca1-36ca9ba02fce None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:11.982353 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4aca69d1-85c1-461f-9ca1-36ca9ba02fce None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:11.995836 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 234/476] 10.222.0.27 () {56 vars in 1109 bytes} [Tue Aug 30 01:43:11 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:43:12.001990 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d4892a8-cd03-4c5a-b778-c184eb49e91a None None] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:12.115178 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d4892a8-cd03-4c5a-b778-c184eb49e91a None None] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:12.160710 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb1e9785-8747-4e6a-ba79-151652caa4e2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:12.161234 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb1e9785-8747-4e6a-ba79-151652caa4e2 None admin] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:12.161587 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb1e9785-8747-4e6a-ba79-151652caa4e2 None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:12.161854 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb1e9785-8747-4e6a-ba79-151652caa4e2 None admin] PATH_INFO: `/v3/users` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:12.162498 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb1e9785-8747-4e6a-ba79-151652caa4e2 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:12.164449 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb1e9785-8747-4e6a-ba79-151652caa4e2 None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:12.204775 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4aca69d1-85c1-461f-9ca1-36ca9ba02fce None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:12.281027 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 243/477] 10.222.0.27 () {60 vars in 1132 bytes} [Tue Aug 30 01:43:12 2022] POST /identity/v3/users => generated 249 bytes in 280 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 01:43:12.329799 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4aca69d1-85c1-461f-9ca1-36ca9ba02fce None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:12.349698 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:43:12.373231 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6399632e-0d2e-42ba-918b-d48e02053a78 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:12.373892 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6399632e-0d2e-42ba-918b-d48e02053a78 None admin] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:12.374177 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6399632e-0d2e-42ba-918b-d48e02053a78 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:12.374444 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6399632e-0d2e-42ba-918b-d48e02053a78 None admin] PATH_INFO: `/v3/roles` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:12.375159 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6399632e-0d2e-42ba-918b-d48e02053a78 None admin] RBAC: Authorizing `identity:create_role()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:12.376686 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6399632e-0d2e-42ba-918b-d48e02053a78 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:12.539892 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.sql.core [None req-6399632e-0d2e-42ba-918b-d48e02053a78 None admin] Conflict role: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'anotherrole-<>' for key 'role.ixu_role_name_domain_id'") Aug 30 01:43:12.539892 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [SQL: INSERT INTO `role` (id, name, description, extra) VALUES (%(id)s, %(name)s, %(description)s, %(extra)s)] Aug 30 01:43:12.539892 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [parameters: {'id': 'a496d9ce147744cbad909b830540f0b0', 'name': 'anotherrole', 'description': None, 'extra': '{}'}] Aug 30 01:43:12.539892 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: (Background on this error at: https://sqlalche.me/e/14/gkpj) {{(pid=81577) wrapper /opt/stack/keystone/keystone/common/sql/core.py:533}} Aug 30 01:43:12.557009 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application [None req-6399632e-0d2e-42ba-918b-d48e02053a78 None admin] Conflict occurred attempting to store role - Duplicate entry found with name anotherrole.: keystone.exception.Conflict: Conflict occurred attempting to store role - Duplicate entry found with name anotherrole. Aug 30 01:43:12.557009 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 01:43:12.557009 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Aug 30 01:43:12.557009 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application self.dialect.do_execute( Aug 30 01:43:12.557009 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Aug 30 01:43:12.557009 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Aug 30 01:43:12.557009 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Aug 30 01:43:12.557009 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application result = self._query(query) Aug 30 01:43:12.557009 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Aug 30 01:43:12.557009 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application conn.query(q) Aug 30 01:43:12.557009 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Aug 30 01:43:12.557009 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Aug 30 01:43:12.557009 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Aug 30 01:43:12.557009 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application result.read() Aug 30 01:43:12.557009 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Aug 30 01:43:12.557997 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Aug 30 01:43:12.557997 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Aug 30 01:43:12.557997 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application packet.raise_for_error() Aug 30 01:43:12.557997 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Aug 30 01:43:12.557997 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Aug 30 01:43:12.557997 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Aug 30 01:43:12.557997 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Aug 30 01:43:12.557997 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application pymysql.err.IntegrityError: (1062, "Duplicate entry 'anotherrole-<>' for key 'role.ixu_role_name_domain_id'") Aug 30 01:43:12.557997 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Aug 30 01:43:12.557997 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application The above exception was the direct cause of the following exception: Aug 30 01:43:12.557997 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Aug 30 01:43:12.557997 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 01:43:12.557997 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 528, in wrapper Aug 30 01:43:12.557997 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return method(*args, **kwargs) Aug 30 01:43:12.557997 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 33, in create_role Aug 30 01:43:12.557997 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return ref.to_dict() Aug 30 01:43:12.557997 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Aug 30 01:43:12.557997 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application next(self.gen) Aug 30 01:43:12.559462 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 1060, in _transaction_scope Aug 30 01:43:12.559462 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application yield resource Aug 30 01:43:12.559462 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Aug 30 01:43:12.559462 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application next(self.gen) Aug 30 01:43:12.559462 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 662, in _session Aug 30 01:43:12.559462 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application self.session.rollback() Aug 30 01:43:12.559462 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Aug 30 01:43:12.559462 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application self.force_reraise() Aug 30 01:43:12.559462 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Aug 30 01:43:12.559462 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise self.value Aug 30 01:43:12.559462 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 659, in _session Aug 30 01:43:12.559462 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application self._end_session_transaction(self.session) Aug 30 01:43:12.559462 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 687, in _end_session_transaction Aug 30 01:43:12.559462 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application session.commit() Aug 30 01:43:12.559462 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 1451, in commit Aug 30 01:43:12.559462 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application self._transaction.commit(_to_root=self.future) Aug 30 01:43:12.559462 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 829, in commit Aug 30 01:43:12.560347 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application self._prepare_impl() Aug 30 01:43:12.560347 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 808, in _prepare_impl Aug 30 01:43:12.560347 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application self.session.flush() Aug 30 01:43:12.560347 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3383, in flush Aug 30 01:43:12.560347 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application self._flush(objects) Aug 30 01:43:12.560347 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3523, in _flush Aug 30 01:43:12.560347 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application transaction.rollback(_capture_exception=True) Aug 30 01:43:12.560347 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__ Aug 30 01:43:12.560347 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application compat.raise_( Aug 30 01:43:12.560347 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Aug 30 01:43:12.560347 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise exception Aug 30 01:43:12.560347 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3483, in _flush Aug 30 01:43:12.560347 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application flush_context.execute() Aug 30 01:43:12.560347 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 456, in execute Aug 30 01:43:12.560347 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application rec.execute(self) Aug 30 01:43:12.560347 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 630, in execute Aug 30 01:43:12.560347 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application util.preloaded.orm_persistence.save_obj( Aug 30 01:43:12.561211 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 245, in save_obj Aug 30 01:43:12.561211 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application _emit_insert_statements( Aug 30 01:43:12.561211 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 1097, in _emit_insert_statements Aug 30 01:43:12.561211 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application c = connection._execute_20( Aug 30 01:43:12.561211 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1705, in _execute_20 Aug 30 01:43:12.561211 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return meth(self, args_10style, kwargs_10style, execution_options) Aug 30 01:43:12.561211 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/sql/elements.py", line 333, in _execute_on_connection Aug 30 01:43:12.561211 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return connection._execute_clauseelement( Aug 30 01:43:12.561211 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1572, in _execute_clauseelement Aug 30 01:43:12.561211 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application ret = self._execute_context( Aug 30 01:43:12.561211 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1943, in _execute_context Aug 30 01:43:12.561211 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application self._handle_dbapi_exception( Aug 30 01:43:12.561211 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 2122, in _handle_dbapi_exception Aug 30 01:43:12.561211 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application util.raise_(newraise, with_traceback=exc_info[2], from_=e) Aug 30 01:43:12.561211 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Aug 30 01:43:12.561211 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise exception Aug 30 01:43:12.563696 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Aug 30 01:43:12.563696 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application self.dialect.do_execute( Aug 30 01:43:12.563696 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Aug 30 01:43:12.563696 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Aug 30 01:43:12.563696 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Aug 30 01:43:12.563696 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application result = self._query(query) Aug 30 01:43:12.563696 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Aug 30 01:43:12.563696 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application conn.query(q) Aug 30 01:43:12.563696 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Aug 30 01:43:12.563696 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Aug 30 01:43:12.563696 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Aug 30 01:43:12.563696 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application result.read() Aug 30 01:43:12.563696 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Aug 30 01:43:12.563696 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Aug 30 01:43:12.563696 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Aug 30 01:43:12.563696 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application packet.raise_for_error() Aug 30 01:43:12.565808 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Aug 30 01:43:12.565808 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Aug 30 01:43:12.565808 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Aug 30 01:43:12.565808 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Aug 30 01:43:12.565808 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application oslo_db.exception.DBDuplicateEntry: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'anotherrole-<>' for key 'role.ixu_role_name_domain_id'") Aug 30 01:43:12.565808 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application [SQL: INSERT INTO `role` (id, name, description, extra) VALUES (%(id)s, %(name)s, %(description)s, %(extra)s)] Aug 30 01:43:12.565808 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application [parameters: {'id': 'a496d9ce147744cbad909b830540f0b0', 'name': 'anotherrole', 'description': None, 'extra': '{}'}] Aug 30 01:43:12.565808 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application (Background on this error at: https://sqlalche.me/e/14/gkpj) Aug 30 01:43:12.565808 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Aug 30 01:43:12.565808 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 30 01:43:12.565808 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Aug 30 01:43:12.565808 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 01:43:12.565808 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 01:43:12.565808 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 01:43:12.565808 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 01:43:12.565808 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 01:43:12.570994 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 01:43:12.570994 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 01:43:12.570994 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 01:43:12.570994 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 01:43:12.570994 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 01:43:12.570994 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 01:43:12.570994 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 115, in post Aug 30 01:43:12.570994 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application ref = PROVIDERS.role_api.create_role( Aug 30 01:43:12.570994 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 01:43:12.570994 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 01:43:12.570994 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1298, in create_role Aug 30 01:43:12.570994 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application ret = self.driver.create_role(role_id, role) Aug 30 01:43:12.570994 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 563, in wrapper Aug 30 01:43:12.570994 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise exception.Conflict(type=conflict_type, Aug 30 01:43:12.570994 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application keystone.exception.Conflict: Conflict occurred attempting to store role - Duplicate entry found with name anotherrole. Aug 30 01:43:12.570994 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Aug 30 01:43:12.574414 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 235/478] 10.222.0.27 () {60 vars in 1131 bytes} [Tue Aug 30 01:43:12 2022] POST /identity/v3/roles => generated 144 bytes in 354 msecs (HTTP/1.1 409) 5 headers in 183 bytes (2 switches on core 0) Aug 30 01:43:12.574549 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb1e9785-8747-4e6a-ba79-151652caa4e2 None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:12.589997 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:43:12.714288 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb1e9785-8747-4e6a-ba79-151652caa4e2 None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:12.797511 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5f25d07-634f-4944-9313-2eaf9362955f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:12.798169 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5f25d07-634f-4944-9313-2eaf9362955f None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:12.798446 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5f25d07-634f-4944-9313-2eaf9362955f None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:12.798714 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5f25d07-634f-4944-9313-2eaf9362955f None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:12.814250 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5f25d07-634f-4944-9313-2eaf9362955f None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 30 01:43:12.819686 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application [None req-a5f25d07-634f-4944-9313-2eaf9362955f None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Aug 30 01:43:12.819686 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 01:43:12.819686 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 01:43:12.819686 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 01:43:12.819686 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 01:43:12.819686 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 01:43:12.819686 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 01:43:12.819686 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 01:43:12.819686 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 01:43:12.819686 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 01:43:12.819686 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 01:43:12.819686 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 01:43:12.819686 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 30 01:43:12.819686 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 30 01:43:12.819686 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 30 01:43:12.819686 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 30 01:43:12.820572 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 30 01:43:12.820572 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 30 01:43:12.820572 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 30 01:43:12.820572 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 30 01:43:12.820572 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 01:43:12.820572 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 01:43:12.820572 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 01:43:12.820572 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 01:43:12.820572 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 01:43:12.820572 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 01:43:12.820572 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 01:43:12.820572 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application with Lock( Aug 30 01:43:12.820572 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 01:43:12.820572 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._enter() Aug 30 01:43:12.820572 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 01:43:12.820572 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 01:43:12.821832 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 01:43:12.821832 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.creator() Aug 30 01:43:12.821832 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 01:43:12.821832 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application created_value = creator( Aug 30 01:43:12.821832 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 30 01:43:12.821832 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 30 01:43:12.821832 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 30 01:43:12.821832 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 30 01:43:12.821832 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 30 01:43:12.821832 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 30 01:43:12.821832 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Aug 30 01:43:12.821832 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application Aug 30 01:43:12.821832 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 244/479] 10.222.0.27 () {56 vars in 1114 bytes} [Tue Aug 30 01:43:12 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 254 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 01:43:12.826666 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6399632e-0d2e-42ba-918b-d48e02053a78 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:12.951457 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6399632e-0d2e-42ba-918b-d48e02053a78 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:12.987314 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-542afad6-c741-4ce2-8c2d-e003b4e12ebc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:12.987921 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-542afad6-c741-4ce2-8c2d-e003b4e12ebc None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:12.988202 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-542afad6-c741-4ce2-8c2d-e003b4e12ebc None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:12.988472 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-542afad6-c741-4ce2-8c2d-e003b4e12ebc None admin] PATH_INFO: `/v3/roles` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:12.989152 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-542afad6-c741-4ce2-8c2d-e003b4e12ebc None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:43:12.989473 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-542afad6-c741-4ce2-8c2d-e003b4e12ebc None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:12.990869 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-542afad6-c741-4ce2-8c2d-e003b4e12ebc None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:13.001670 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 236/480] 10.222.0.27 () {56 vars in 1111 bytes} [Tue Aug 30 01:43:12 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:43:13.760896 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc8ba480-7850-413f-a44a-9b3cea269d8e None None] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:13.761573 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc8ba480-7850-413f-a44a-9b3cea269d8e None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:13.761573 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc8ba480-7850-413f-a44a-9b3cea269d8e None None] PATH_INFO: `/` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:13.763121 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 245/481] 10.222.0.27 () {54 vars in 927 bytes} [Tue Aug 30 01:43:13 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 01:43:13.767941 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a739b939-e78b-4314-831a-4d6ba79fe533 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:13.768403 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a739b939-e78b-4314-831a-4d6ba79fe533 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:13.768569 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a739b939-e78b-4314-831a-4d6ba79fe533 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:13.790620 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-a739b939-e78b-4314-831a-4d6ba79fe533 None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 01:43:13.791838 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-a739b939-e78b-4314-831a-4d6ba79fe533 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:43:13.845321 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:43:13.885998 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 237/482] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Aug 30 01:43:13 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 119 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 30 01:43:14.021627 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bdced8e-2505-4305-b6dd-80f2896083eb None None] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:14.021960 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bdced8e-2505-4305-b6dd-80f2896083eb None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:14.022129 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bdced8e-2505-4305-b6dd-80f2896083eb None None] PATH_INFO: `/` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:14.024522 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 246/483] 10.222.0.27 () {54 vars in 927 bytes} [Tue Aug 30 01:43:14 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 01:43:14.029981 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0880052f-39a6-4ee1-8e88-aa3591ed499d None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:14.030270 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0880052f-39a6-4ee1-8e88-aa3591ed499d None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:14.030552 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0880052f-39a6-4ee1-8e88-aa3591ed499d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:14.050146 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-0880052f-39a6-4ee1-8e88-aa3591ed499d None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 01:43:14.051350 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-0880052f-39a6-4ee1-8e88-aa3591ed499d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:43:14.123376 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 238/484] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Aug 30 01:43:14 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 30 01:43:14.255828 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bdced8e-2505-4305-b6dd-80f2896083eb None None] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:14.291485 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91509c34-99ba-45fc-9fa6-4d12d91edfed None None] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:14.291771 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91509c34-99ba-45fc-9fa6-4d12d91edfed None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:14.291965 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91509c34-99ba-45fc-9fa6-4d12d91edfed None None] PATH_INFO: `/` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:14.293570 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 239/485] 10.222.0.27 () {54 vars in 927 bytes} [Tue Aug 30 01:43:14 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 01:43:14.298646 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a835a940-f99f-4edf-9de2-9afb0d55c828 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:14.298934 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a835a940-f99f-4edf-9de2-9afb0d55c828 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:14.299215 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a835a940-f99f-4edf-9de2-9afb0d55c828 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:14.318901 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-a835a940-f99f-4edf-9de2-9afb0d55c828 None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 01:43:14.321055 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-a835a940-f99f-4edf-9de2-9afb0d55c828 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:43:14.370059 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:43:14.427149 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bdced8e-2505-4305-b6dd-80f2896083eb None None] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:14.452925 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 240/486] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Aug 30 01:43:14 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 155 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Aug 30 01:43:14.463125 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a835a940-f99f-4edf-9de2-9afb0d55c828 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:14.498062 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77cd96f8-8ae7-42c3-a82d-a3c4ae7e5aef None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:14.498692 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77cd96f8-8ae7-42c3-a82d-a3c4ae7e5aef None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:14.498969 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77cd96f8-8ae7-42c3-a82d-a3c4ae7e5aef None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:14.499237 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77cd96f8-8ae7-42c3-a82d-a3c4ae7e5aef None admin] PATH_INFO: `/v3/roles/service` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:14.517943 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77cd96f8-8ae7-42c3-a82d-a3c4ae7e5aef None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 30 01:43:14.524243 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application [None req-77cd96f8-8ae7-42c3-a82d-a3c4ae7e5aef None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Aug 30 01:43:14.524243 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 01:43:14.524243 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 01:43:14.524243 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 01:43:14.524243 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 01:43:14.524243 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 01:43:14.524243 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 01:43:14.524243 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 01:43:14.524243 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 01:43:14.524243 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 01:43:14.524243 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 01:43:14.524243 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 01:43:14.524243 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 30 01:43:14.524243 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 30 01:43:14.524243 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 30 01:43:14.524243 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 30 01:43:14.525144 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 30 01:43:14.525144 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 30 01:43:14.525144 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 30 01:43:14.525144 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 30 01:43:14.525144 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 01:43:14.525144 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 01:43:14.525144 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 01:43:14.525144 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 01:43:14.525144 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 01:43:14.525144 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 01:43:14.525144 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 01:43:14.525144 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application with Lock( Aug 30 01:43:14.525144 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 01:43:14.525144 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._enter() Aug 30 01:43:14.525144 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 01:43:14.525144 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 01:43:14.526362 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 01:43:14.526362 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.creator() Aug 30 01:43:14.526362 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 01:43:14.526362 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application created_value = creator( Aug 30 01:43:14.526362 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 30 01:43:14.526362 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 30 01:43:14.526362 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 30 01:43:14.526362 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 30 01:43:14.526362 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 30 01:43:14.526362 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 30 01:43:14.526362 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Aug 30 01:43:14.526362 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application Aug 30 01:43:14.526362 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 247/487] 10.222.0.27 () {56 vars in 1102 bytes} [Tue Aug 30 01:43:14 2022] GET /identity/v3/roles/service => generated 85 bytes in 270 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 01:43:14.529801 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:43:14.536746 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77cd96f8-8ae7-42c3-a82d-a3c4ae7e5aef None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:14.653909 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a835a940-f99f-4edf-9de2-9afb0d55c828 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:14.665252 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77cd96f8-8ae7-42c3-a82d-a3c4ae7e5aef None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:14.692218 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48ff47a4-29af-4eba-b5c0-db1eafaf6879 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:14.693011 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48ff47a4-29af-4eba-b5c0-db1eafaf6879 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:14.693394 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48ff47a4-29af-4eba-b5c0-db1eafaf6879 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:14.693793 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48ff47a4-29af-4eba-b5c0-db1eafaf6879 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:14.708511 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48ff47a4-29af-4eba-b5c0-db1eafaf6879 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 30 01:43:14.714476 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application [None req-48ff47a4-29af-4eba-b5c0-db1eafaf6879 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Aug 30 01:43:14.714476 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 01:43:14.714476 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 01:43:14.714476 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 01:43:14.714476 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 01:43:14.714476 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 01:43:14.714476 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 01:43:14.714476 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 01:43:14.714476 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 01:43:14.714476 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 01:43:14.714476 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 01:43:14.714476 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 01:43:14.714476 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 30 01:43:14.714476 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 30 01:43:14.714476 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 30 01:43:14.714476 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 30 01:43:14.715602 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 30 01:43:14.715602 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 30 01:43:14.715602 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 30 01:43:14.715602 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 30 01:43:14.715602 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 01:43:14.715602 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 01:43:14.715602 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 01:43:14.715602 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 01:43:14.715602 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 01:43:14.715602 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 01:43:14.715602 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 01:43:14.715602 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application with Lock( Aug 30 01:43:14.715602 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 01:43:14.715602 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._enter() Aug 30 01:43:14.715602 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 01:43:14.715602 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 01:43:14.716626 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 01:43:14.716626 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.creator() Aug 30 01:43:14.716626 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 01:43:14.716626 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application created_value = creator( Aug 30 01:43:14.716626 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 30 01:43:14.716626 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 30 01:43:14.716626 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 30 01:43:14.716626 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 30 01:43:14.716626 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 30 01:43:14.716626 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 30 01:43:14.716626 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Aug 30 01:43:14.716626 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Aug 30 01:43:14.720702 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 241/488] 10.222.0.27 () {56 vars in 1102 bytes} [Tue Aug 30 01:43:14 2022] GET /identity/v3/roles/service => generated 85 bytes in 256 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 30 01:43:14.720702 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-118b7ea8-b799-473a-aa73-5f32f210b189 None None] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:14.720702 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-118b7ea8-b799-473a-aa73-5f32f210b189 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:14.720702 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-118b7ea8-b799-473a-aa73-5f32f210b189 None None] PATH_INFO: `/` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:14.724528 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c769297-8025-4c6a-9bb2-b76d97a76434 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:14.724996 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 242/489] 10.222.0.27 () {54 vars in 927 bytes} [Tue Aug 30 01:43:14 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 01:43:14.725591 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c769297-8025-4c6a-9bb2-b76d97a76434 None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:14.726266 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c769297-8025-4c6a-9bb2-b76d97a76434 None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:14.726699 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-118b7ea8-b799-473a-aa73-5f32f210b189 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:14.727339 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c769297-8025-4c6a-9bb2-b76d97a76434 None admin] PATH_INFO: `/v3/roles` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:14.728659 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c769297-8025-4c6a-9bb2-b76d97a76434 None admin] RBAC: Adding query filter params (name=service) {{(pid=81578) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:43:14.729283 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c769297-8025-4c6a-9bb2-b76d97a76434 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:14.731566 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c769297-8025-4c6a-9bb2-b76d97a76434 None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:14.734424 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:43:14.745736 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 248/490] 10.222.0.27 () {56 vars in 1103 bytes} [Tue Aug 30 01:43:14 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 214 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:43:14.748424 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3c5734d-a4e2-40d1-8030-429ce6ecde55 None None] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:14.748911 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3c5734d-a4e2-40d1-8030-429ce6ecde55 None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:14.750236 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3c5734d-a4e2-40d1-8030-429ce6ecde55 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:14.774823 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.auth.core [None req-d3c5734d-a4e2-40d1-8030-429ce6ecde55 None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81578) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 01:43:14.775988 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-d3c5734d-a4e2-40d1-8030-429ce6ecde55 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:43:14.840684 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:43:14.847748 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-118b7ea8-b799-473a-aa73-5f32f210b189 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:14.854823 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 249/491] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Aug 30 01:43:14 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 30 01:43:14.856276 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3c5734d-a4e2-40d1-8030-429ce6ecde55 None None] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:14.882070 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1eb6908-e5f7-4b2a-8b62-5e235c5124ec None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:14.882746 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1eb6908-e5f7-4b2a-8b62-5e235c5124ec None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:14.883022 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1eb6908-e5f7-4b2a-8b62-5e235c5124ec None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:14.883293 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1eb6908-e5f7-4b2a-8b62-5e235c5124ec None admin] PATH_INFO: `/v3/roles` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:14.883950 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1eb6908-e5f7-4b2a-8b62-5e235c5124ec None admin] RBAC: Adding query filter params (name=service) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:43:14.884247 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1eb6908-e5f7-4b2a-8b62-5e235c5124ec None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:14.885643 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1eb6908-e5f7-4b2a-8b62-5e235c5124ec None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:14.893583 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 243/492] 10.222.0.27 () {56 vars in 1103 bytes} [Tue Aug 30 01:43:14 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:43:14.894447 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1eb6908-e5f7-4b2a-8b62-5e235c5124ec None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:14.984662 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3c5734d-a4e2-40d1-8030-429ce6ecde55 None None] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:15.012887 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:43:15.013400 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1eb6908-e5f7-4b2a-8b62-5e235c5124ec None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:15.022270 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-889a0d1d-dd66-4c61-9a39-cef6ccffbefd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:15.023864 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-889a0d1d-dd66-4c61-9a39-cef6ccffbefd None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:15.029951 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-889a0d1d-dd66-4c61-9a39-cef6ccffbefd None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:15.029951 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-889a0d1d-dd66-4c61-9a39-cef6ccffbefd None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:15.035350 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-889a0d1d-dd66-4c61-9a39-cef6ccffbefd None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:15.036990 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-889a0d1d-dd66-4c61-9a39-cef6ccffbefd None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:15.050354 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88d430e9-9fb7-48f7-8281-3570467ae972 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:15.051028 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88d430e9-9fb7-48f7-8281-3570467ae972 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:15.051306 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88d430e9-9fb7-48f7-8281-3570467ae972 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:15.051575 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88d430e9-9fb7-48f7-8281-3570467ae972 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:15.055536 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application [None req-889a0d1d-dd66-4c61-9a39-cef6ccffbefd None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 01:43:15.055536 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 01:43:15.055536 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 01:43:15.055536 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 01:43:15.055536 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 01:43:15.055536 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 01:43:15.055536 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 01:43:15.055536 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 01:43:15.055536 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 01:43:15.055536 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 01:43:15.055536 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 01:43:15.055536 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 01:43:15.055536 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 30 01:43:15.055536 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 30 01:43:15.055536 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 30 01:43:15.055536 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 30 01:43:15.056400 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 01:43:15.056400 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 01:43:15.056400 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 01:43:15.056400 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 01:43:15.056400 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 01:43:15.056400 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 01:43:15.056400 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 01:43:15.056400 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application with Lock( Aug 30 01:43:15.056400 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 01:43:15.056400 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._enter() Aug 30 01:43:15.056400 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 01:43:15.056400 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 01:43:15.056400 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 01:43:15.056400 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.creator() Aug 30 01:43:15.056400 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 01:43:15.056400 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application created_value = creator( Aug 30 01:43:15.056400 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 30 01:43:15.057225 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 30 01:43:15.057225 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 01:43:15.057225 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application Aug 30 01:43:15.058798 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 250/493] 10.222.0.27 () {56 vars in 1108 bytes} [Tue Aug 30 01:43:14 2022] GET /identity/v3/domains/Default => generated 87 bytes in 203 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 01:43:15.059748 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-889a0d1d-dd66-4c61-9a39-cef6ccffbefd None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:15.068415 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88d430e9-9fb7-48f7-8281-3570467ae972 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 30 01:43:15.075250 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application [None req-88d430e9-9fb7-48f7-8281-3570467ae972 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Aug 30 01:43:15.075250 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 01:43:15.075250 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 01:43:15.075250 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 01:43:15.075250 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 01:43:15.075250 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 01:43:15.075250 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 01:43:15.075250 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 01:43:15.075250 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 01:43:15.075250 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 01:43:15.075250 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 01:43:15.075250 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 01:43:15.075250 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 30 01:43:15.075250 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 30 01:43:15.075250 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 30 01:43:15.075250 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 30 01:43:15.076152 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 30 01:43:15.076152 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 30 01:43:15.076152 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 30 01:43:15.076152 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 30 01:43:15.076152 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 01:43:15.076152 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 01:43:15.076152 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 01:43:15.076152 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 01:43:15.076152 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 01:43:15.076152 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 01:43:15.076152 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 01:43:15.076152 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application with Lock( Aug 30 01:43:15.076152 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 01:43:15.076152 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._enter() Aug 30 01:43:15.076152 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 01:43:15.076152 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 01:43:15.076963 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 01:43:15.076963 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.creator() Aug 30 01:43:15.076963 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 01:43:15.076963 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application created_value = creator( Aug 30 01:43:15.076963 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 30 01:43:15.076963 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 30 01:43:15.076963 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 30 01:43:15.076963 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 30 01:43:15.076963 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 30 01:43:15.076963 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 30 01:43:15.076963 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Aug 30 01:43:15.076963 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Aug 30 01:43:15.081220 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 244/494] 10.222.0.27 () {56 vars in 1102 bytes} [Tue Aug 30 01:43:14 2022] GET /identity/v3/roles/service => generated 85 bytes in 183 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 01:43:15.082244 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88d430e9-9fb7-48f7-8281-3570467ae972 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:15.103552 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:43:15.168512 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-889a0d1d-dd66-4c61-9a39-cef6ccffbefd None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:15.192137 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88d430e9-9fb7-48f7-8281-3570467ae972 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:15.204770 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4adb0fb2-6fc5-44b4-b010-c9b104904ce1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:15.205393 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4adb0fb2-6fc5-44b4-b010-c9b104904ce1 None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:15.205848 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4adb0fb2-6fc5-44b4-b010-c9b104904ce1 None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:15.206114 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4adb0fb2-6fc5-44b4-b010-c9b104904ce1 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:15.217816 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4adb0fb2-6fc5-44b4-b010-c9b104904ce1 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:15.219260 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4adb0fb2-6fc5-44b4-b010-c9b104904ce1 None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:15.228003 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03f6f23b-fb9b-44af-ab5a-78a59a006854 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:15.228719 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03f6f23b-fb9b-44af-ab5a-78a59a006854 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:15.228999 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03f6f23b-fb9b-44af-ab5a-78a59a006854 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:15.229360 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03f6f23b-fb9b-44af-ab5a-78a59a006854 None admin] PATH_INFO: `/v3/domains` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:15.229936 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03f6f23b-fb9b-44af-ab5a-78a59a006854 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:43:15.230239 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03f6f23b-fb9b-44af-ab5a-78a59a006854 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:15.231601 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03f6f23b-fb9b-44af-ab5a-78a59a006854 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:15.234493 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application [None req-4adb0fb2-6fc5-44b4-b010-c9b104904ce1 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 01:43:15.234493 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 01:43:15.234493 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 01:43:15.234493 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 01:43:15.234493 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 01:43:15.234493 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 01:43:15.234493 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 01:43:15.234493 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 01:43:15.234493 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 01:43:15.234493 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 01:43:15.234493 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 01:43:15.234493 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 01:43:15.234493 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 30 01:43:15.234493 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 30 01:43:15.234493 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 30 01:43:15.234493 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 30 01:43:15.235441 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 01:43:15.235441 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 01:43:15.235441 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 01:43:15.235441 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 01:43:15.235441 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 01:43:15.235441 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 01:43:15.235441 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 01:43:15.235441 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application with Lock( Aug 30 01:43:15.235441 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 01:43:15.235441 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._enter() Aug 30 01:43:15.235441 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 01:43:15.235441 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 01:43:15.235441 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 01:43:15.235441 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.creator() Aug 30 01:43:15.235441 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 01:43:15.235441 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application created_value = creator( Aug 30 01:43:15.235441 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 30 01:43:15.236305 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 30 01:43:15.236305 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 01:43:15.236305 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application Aug 30 01:43:15.236305 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 251/495] 10.222.0.27 () {56 vars in 1108 bytes} [Tue Aug 30 01:43:15 2022] GET /identity/v3/domains/Default => generated 87 bytes in 176 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 01:43:15.237022 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4adb0fb2-6fc5-44b4-b010-c9b104904ce1 None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:15.253186 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 245/496] 10.222.0.27 () {56 vars in 1109 bytes} [Tue Aug 30 01:43:15 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:43:15.253186 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03f6f23b-fb9b-44af-ab5a-78a59a006854 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:15.277234 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:43:15.331235 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:43:15.351548 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4adb0fb2-6fc5-44b4-b010-c9b104904ce1 None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:15.354694 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03f6f23b-fb9b-44af-ab5a-78a59a006854 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:15.390252 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9443fbcb-625b-4e14-833f-8aedd37777af None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:15.390853 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9443fbcb-625b-4e14-833f-8aedd37777af None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:15.391132 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9443fbcb-625b-4e14-833f-8aedd37777af None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:15.391360 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd2bd518-284e-47ea-b74e-9020ea87b0ac None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:15.391472 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9443fbcb-625b-4e14-833f-8aedd37777af None admin] PATH_INFO: `/v3/roles` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:15.392050 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9443fbcb-625b-4e14-833f-8aedd37777af None admin] RBAC: Adding query filter params (name=service) {{(pid=81578) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:43:15.392352 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9443fbcb-625b-4e14-833f-8aedd37777af None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:15.392613 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd2bd518-284e-47ea-b74e-9020ea87b0ac None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:15.393074 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd2bd518-284e-47ea-b74e-9020ea87b0ac None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:15.393555 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd2bd518-284e-47ea-b74e-9020ea87b0ac None admin] PATH_INFO: `/v3/roles/service` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:15.393777 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9443fbcb-625b-4e14-833f-8aedd37777af None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:15.402374 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 252/497] 10.222.0.27 () {56 vars in 1103 bytes} [Tue Aug 30 01:43:15 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:43:15.403167 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9443fbcb-625b-4e14-833f-8aedd37777af None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:15.404882 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd2bd518-284e-47ea-b74e-9020ea87b0ac None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 30 01:43:15.410219 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application [None req-cd2bd518-284e-47ea-b74e-9020ea87b0ac None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Aug 30 01:43:15.410219 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 01:43:15.410219 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 01:43:15.410219 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 01:43:15.410219 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 01:43:15.410219 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 01:43:15.410219 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 01:43:15.410219 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 01:43:15.410219 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 01:43:15.410219 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 01:43:15.410219 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 01:43:15.410219 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 01:43:15.410219 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 30 01:43:15.410219 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 30 01:43:15.410219 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 30 01:43:15.410219 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 30 01:43:15.411113 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 30 01:43:15.411113 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 30 01:43:15.411113 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 30 01:43:15.411113 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 30 01:43:15.411113 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 01:43:15.411113 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 01:43:15.411113 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 01:43:15.411113 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 01:43:15.411113 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 01:43:15.411113 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 01:43:15.411113 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 01:43:15.411113 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application with Lock( Aug 30 01:43:15.411113 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 01:43:15.411113 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._enter() Aug 30 01:43:15.411113 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 01:43:15.411113 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 01:43:15.412024 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 01:43:15.412024 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.creator() Aug 30 01:43:15.412024 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 01:43:15.412024 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application created_value = creator( Aug 30 01:43:15.412024 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 30 01:43:15.412024 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 30 01:43:15.412024 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 30 01:43:15.412024 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 30 01:43:15.412024 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 30 01:43:15.412024 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 30 01:43:15.412024 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Aug 30 01:43:15.412024 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Aug 30 01:43:15.415970 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 246/498] 10.222.0.27 () {56 vars in 1102 bytes} [Tue Aug 30 01:43:15 2022] GET /identity/v3/roles/service => generated 85 bytes in 167 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 01:43:15.416803 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd2bd518-284e-47ea-b74e-9020ea87b0ac None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:15.513557 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:43:15.522198 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9443fbcb-625b-4e14-833f-8aedd37777af None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:15.529780 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd2bd518-284e-47ea-b74e-9020ea87b0ac None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:15.557844 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7ea0140-8ce9-4535-ba67-e0a02084eb5d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:15.558454 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7ea0140-8ce9-4535-ba67-e0a02084eb5d None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:15.558744 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7ea0140-8ce9-4535-ba67-e0a02084eb5d None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:15.559016 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7ea0140-8ce9-4535-ba67-e0a02084eb5d None admin] PATH_INFO: `/v3/domains` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:15.559663 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7ea0140-8ce9-4535-ba67-e0a02084eb5d None admin] RBAC: Adding query filter params (name=Default) {{(pid=81578) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:43:15.559973 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7ea0140-8ce9-4535-ba67-e0a02084eb5d None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:15.561330 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7ea0140-8ce9-4535-ba67-e0a02084eb5d None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:15.569238 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d2b42ba-94b7-49ab-9453-3d5d0ba93f99 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:15.570594 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d2b42ba-94b7-49ab-9453-3d5d0ba93f99 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:15.571320 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d2b42ba-94b7-49ab-9453-3d5d0ba93f99 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:15.571641 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d2b42ba-94b7-49ab-9453-3d5d0ba93f99 None admin] PATH_INFO: `/v3/users/glance` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:15.578652 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 253/499] 10.222.0.27 () {56 vars in 1109 bytes} [Tue Aug 30 01:43:15 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:43:15.578652 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7ea0140-8ce9-4535-ba67-e0a02084eb5d None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:15.579461 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:43:15.580473 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d2b42ba-94b7-49ab-9453-3d5d0ba93f99 None admin] RBAC: Authorizing `identity:get_user(user_id=glance)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:15.583179 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d2b42ba-94b7-49ab-9453-3d5d0ba93f99 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:15.599068 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application [None req-9d2b42ba-94b7-49ab-9453-3d5d0ba93f99 None admin] Could not find user: glance.: keystone.exception.UserNotFound: Could not find user: glance. Aug 30 01:43:15.599068 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 01:43:15.599068 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 01:43:15.599068 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 01:43:15.599068 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 01:43:15.599068 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 01:43:15.599068 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 01:43:15.599068 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 01:43:15.599068 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 01:43:15.599068 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 01:43:15.599068 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 01:43:15.599068 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 01:43:15.599068 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 30 01:43:15.599068 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 30 01:43:15.599068 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 30 01:43:15.599068 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 30 01:43:15.600192 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 01:43:15.600192 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 01:43:15.600192 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 30 01:43:15.600192 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 30 01:43:15.600192 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 30 01:43:15.600192 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 30 01:43:15.600192 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 01:43:15.600192 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 01:43:15.600192 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 01:43:15.600192 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 01:43:15.600192 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 01:43:15.600192 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application with Lock( Aug 30 01:43:15.600192 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 01:43:15.600192 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._enter() Aug 30 01:43:15.600192 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 01:43:15.600192 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 01:43:15.600192 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 01:43:15.600192 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.creator() Aug 30 01:43:15.602651 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 01:43:15.602651 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application created_value = creator( Aug 30 01:43:15.602651 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 30 01:43:15.602651 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 30 01:43:15.602651 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 30 01:43:15.602651 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 30 01:43:15.602651 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 30 01:43:15.602651 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 30 01:43:15.602651 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance. Aug 30 01:43:15.602651 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Aug 30 01:43:15.602651 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 247/500] 10.222.0.27 () {56 vars in 1099 bytes} [Tue Aug 30 01:43:15 2022] GET /identity/v3/users/glance => generated 84 bytes in 186 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 01:43:15.603442 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d2b42ba-94b7-49ab-9453-3d5d0ba93f99 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:15.685268 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7ea0140-8ce9-4535-ba67-e0a02084eb5d None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:15.709169 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d2b42ba-94b7-49ab-9453-3d5d0ba93f99 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:15.720369 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6752008f-97bd-4408-9d8f-671963599a89 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:15.721123 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6752008f-97bd-4408-9d8f-671963599a89 None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:15.721615 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6752008f-97bd-4408-9d8f-671963599a89 None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:15.721963 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6752008f-97bd-4408-9d8f-671963599a89 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:15.732717 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6752008f-97bd-4408-9d8f-671963599a89 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:15.734226 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6752008f-97bd-4408-9d8f-671963599a89 None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:15.744408 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7525508a-6663-4e32-a8d5-686b998ba563 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:15.745098 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7525508a-6663-4e32-a8d5-686b998ba563 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:15.745364 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7525508a-6663-4e32-a8d5-686b998ba563 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:15.745662 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7525508a-6663-4e32-a8d5-686b998ba563 None admin] PATH_INFO: `/v3/roles` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:15.746316 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7525508a-6663-4e32-a8d5-686b998ba563 None admin] RBAC: Adding query filter params (name=service) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:43:15.746615 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7525508a-6663-4e32-a8d5-686b998ba563 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:15.747959 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7525508a-6663-4e32-a8d5-686b998ba563 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:15.749363 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application [None req-6752008f-97bd-4408-9d8f-671963599a89 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 01:43:15.749363 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 01:43:15.749363 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 01:43:15.749363 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 01:43:15.749363 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 01:43:15.749363 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 01:43:15.749363 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 01:43:15.749363 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 01:43:15.749363 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 01:43:15.749363 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 01:43:15.749363 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 01:43:15.749363 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 01:43:15.749363 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 30 01:43:15.749363 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 30 01:43:15.749363 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 30 01:43:15.749363 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 30 01:43:15.750445 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 01:43:15.750445 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 01:43:15.750445 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 01:43:15.750445 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 01:43:15.750445 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 01:43:15.750445 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 01:43:15.750445 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 01:43:15.750445 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application with Lock( Aug 30 01:43:15.750445 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 01:43:15.750445 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._enter() Aug 30 01:43:15.750445 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 01:43:15.750445 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 01:43:15.750445 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 01:43:15.750445 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.creator() Aug 30 01:43:15.750445 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 01:43:15.750445 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application created_value = creator( Aug 30 01:43:15.750445 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 30 01:43:15.751471 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 30 01:43:15.751471 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 01:43:15.751471 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application Aug 30 01:43:15.751471 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 254/501] 10.222.0.27 () {56 vars in 1108 bytes} [Tue Aug 30 01:43:15 2022] GET /identity/v3/domains/Default => generated 87 bytes in 176 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 01:43:15.751471 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6752008f-97bd-4408-9d8f-671963599a89 None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:15.754583 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:43:15.759820 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 248/502] 10.222.0.27 () {56 vars in 1103 bytes} [Tue Aug 30 01:43:15 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:43:15.767171 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f0f07ea-4c6a-4fc3-9813-1cafda9f591e None None] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:15.767171 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f0f07ea-4c6a-4fc3-9813-1cafda9f591e None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:15.767171 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f0f07ea-4c6a-4fc3-9813-1cafda9f591e None None] PATH_INFO: `/` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:15.767171 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 249/503] 10.222.0.27 () {54 vars in 927 bytes} [Tue Aug 30 01:43:15 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 01:43:15.767171 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f0f07ea-4c6a-4fc3-9813-1cafda9f591e None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:15.826328 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:43:15.853050 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6752008f-97bd-4408-9d8f-671963599a89 None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:15.869572 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f0f07ea-4c6a-4fc3-9813-1cafda9f591e None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:15.886403 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75eae3b5-91d3-4e4e-98ab-757ec564858b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:15.887024 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75eae3b5-91d3-4e4e-98ab-757ec564858b None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:15.887307 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75eae3b5-91d3-4e4e-98ab-757ec564858b None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:15.887577 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75eae3b5-91d3-4e4e-98ab-757ec564858b None admin] PATH_INFO: `/v3/users/nova` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:15.895062 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75eae3b5-91d3-4e4e-98ab-757ec564858b None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:15.896477 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75eae3b5-91d3-4e4e-98ab-757ec564858b None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:15.903197 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-816d3327-b9f1-4eb1-afb0-4be6a48ef814 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:15.903788 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-816d3327-b9f1-4eb1-afb0-4be6a48ef814 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:15.904063 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-816d3327-b9f1-4eb1-afb0-4be6a48ef814 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:15.904331 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-816d3327-b9f1-4eb1-afb0-4be6a48ef814 None admin] PATH_INFO: `/v3/users` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:15.905047 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-816d3327-b9f1-4eb1-afb0-4be6a48ef814 None admin] RBAC: Adding query filter params (domain_id=default, name=glance) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:43:15.905354 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-816d3327-b9f1-4eb1-afb0-4be6a48ef814 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:15.906775 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-816d3327-b9f1-4eb1-afb0-4be6a48ef814 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:15.909332 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application [None req-75eae3b5-91d3-4e4e-98ab-757ec564858b None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Aug 30 01:43:15.909332 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 01:43:15.909332 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 01:43:15.909332 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 01:43:15.909332 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 01:43:15.909332 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 01:43:15.909332 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 01:43:15.909332 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 01:43:15.909332 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 01:43:15.909332 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 01:43:15.909332 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 01:43:15.909332 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 01:43:15.909332 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 30 01:43:15.909332 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 30 01:43:15.909332 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 30 01:43:15.909332 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 30 01:43:15.910368 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 01:43:15.910368 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 01:43:15.910368 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 30 01:43:15.910368 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 30 01:43:15.910368 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 30 01:43:15.910368 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 30 01:43:15.910368 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 01:43:15.910368 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 01:43:15.910368 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 01:43:15.910368 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 01:43:15.910368 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 01:43:15.910368 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application with Lock( Aug 30 01:43:15.910368 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 01:43:15.910368 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._enter() Aug 30 01:43:15.910368 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 01:43:15.910368 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 01:43:15.910368 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 01:43:15.910368 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.creator() Aug 30 01:43:15.911716 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 01:43:15.911716 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application created_value = creator( Aug 30 01:43:15.911716 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 30 01:43:15.911716 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 30 01:43:15.911716 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 30 01:43:15.911716 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 30 01:43:15.911716 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 30 01:43:15.911716 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 30 01:43:15.911716 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Aug 30 01:43:15.911716 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application Aug 30 01:43:15.911716 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 255/504] 10.222.0.27 () {56 vars in 1093 bytes} [Tue Aug 30 01:43:15 2022] GET /identity/v3/users/nova => generated 82 bytes in 160 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 01:43:15.911716 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75eae3b5-91d3-4e4e-98ab-757ec564858b None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:15.955678 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 250/505] 10.222.0.27 () {56 vars in 1137 bytes} [Tue Aug 30 01:43:15 2022] GET /identity/v3/users?domain_id=default&name=glance => generated 372 bytes in 189 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:43:15.956519 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-816d3327-b9f1-4eb1-afb0-4be6a48ef814 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:15.974833 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:43:16.012849 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75eae3b5-91d3-4e4e-98ab-757ec564858b None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:16.045640 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-adb648f8-f7f2-48fb-bf73-d7d70e186016 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:16.046239 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adb648f8-f7f2-48fb-bf73-d7d70e186016 None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:16.046530 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adb648f8-f7f2-48fb-bf73-d7d70e186016 None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:16.046783 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adb648f8-f7f2-48fb-bf73-d7d70e186016 None admin] PATH_INFO: `/v3/domains` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:16.047415 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-adb648f8-f7f2-48fb-bf73-d7d70e186016 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81578) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:43:16.047713 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-adb648f8-f7f2-48fb-bf73-d7d70e186016 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:16.049086 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-adb648f8-f7f2-48fb-bf73-d7d70e186016 None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:16.058081 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-816d3327-b9f1-4eb1-afb0-4be6a48ef814 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:16.061235 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 256/506] 10.222.0.27 () {56 vars in 1109 bytes} [Tue Aug 30 01:43:15 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:43:16.062418 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e15d3c15-31f6-419d-91d5-afe8d5e48ab6 None None] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:16.062696 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e15d3c15-31f6-419d-91d5-afe8d5e48ab6 None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:16.062959 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e15d3c15-31f6-419d-91d5-afe8d5e48ab6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:16.075297 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:43:16.080765 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.auth.core [None req-e15d3c15-31f6-419d-91d5-afe8d5e48ab6 None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81578) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 01:43:16.081998 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-e15d3c15-31f6-419d-91d5-afe8d5e48ab6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:43:16.092478 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98ae04f9-aa9e-49b6-8138-65e0faca59ca None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:16.093115 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98ae04f9-aa9e-49b6-8138-65e0faca59ca None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:16.093421 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98ae04f9-aa9e-49b6-8138-65e0faca59ca None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:16.093737 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98ae04f9-aa9e-49b6-8138-65e0faca59ca None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:16.104097 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98ae04f9-aa9e-49b6-8138-65e0faca59ca None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:16.106103 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98ae04f9-aa9e-49b6-8138-65e0faca59ca None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:16.120789 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application [None req-98ae04f9-aa9e-49b6-8138-65e0faca59ca None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 01:43:16.120789 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 01:43:16.120789 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 01:43:16.120789 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 01:43:16.120789 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 01:43:16.120789 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 01:43:16.120789 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 01:43:16.120789 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 01:43:16.120789 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 01:43:16.120789 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 01:43:16.120789 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 01:43:16.120789 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 01:43:16.120789 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 30 01:43:16.120789 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 30 01:43:16.120789 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 30 01:43:16.120789 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 30 01:43:16.121966 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 01:43:16.121966 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 01:43:16.121966 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 01:43:16.121966 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 01:43:16.121966 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 01:43:16.121966 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 01:43:16.121966 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 01:43:16.121966 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application with Lock( Aug 30 01:43:16.121966 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 01:43:16.121966 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._enter() Aug 30 01:43:16.121966 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 01:43:16.121966 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 01:43:16.121966 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 01:43:16.121966 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.creator() Aug 30 01:43:16.121966 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 01:43:16.121966 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application created_value = creator( Aug 30 01:43:16.121966 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 30 01:43:16.122847 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 30 01:43:16.122847 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 01:43:16.122847 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Aug 30 01:43:16.122847 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 251/507] 10.222.0.27 () {56 vars in 1108 bytes} [Tue Aug 30 01:43:15 2022] GET /identity/v3/domains/Default => generated 87 bytes in 166 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 01:43:16.123071 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98ae04f9-aa9e-49b6-8138-65e0faca59ca None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:16.150514 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 257/508] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Aug 30 01:43:16 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 30 01:43:16.151202 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e15d3c15-31f6-419d-91d5-afe8d5e48ab6 None None] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:16.207826 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:43:16.230877 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98ae04f9-aa9e-49b6-8138-65e0faca59ca None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:16.259837 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e15d3c15-31f6-419d-91d5-afe8d5e48ab6 None None] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:16.269621 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d92f9ea-26f9-4f3f-ab51-651890e9082a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:16.270318 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d92f9ea-26f9-4f3f-ab51-651890e9082a None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:16.270514 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d92f9ea-26f9-4f3f-ab51-651890e9082a None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:16.270789 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d92f9ea-26f9-4f3f-ab51-651890e9082a None admin] PATH_INFO: `/v3/users` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:16.271513 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d92f9ea-26f9-4f3f-ab51-651890e9082a None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:43:16.271824 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d92f9ea-26f9-4f3f-ab51-651890e9082a None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:16.273930 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d92f9ea-26f9-4f3f-ab51-651890e9082a None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:16.293670 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 252/509] 10.222.0.27 () {56 vars in 1133 bytes} [Tue Aug 30 01:43:16 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:43:16.294558 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d92f9ea-26f9-4f3f-ab51-651890e9082a None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:16.302740 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35e812c2-7631-4c77-a986-04dbbf1c2b3c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:16.303297 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35e812c2-7631-4c77-a986-04dbbf1c2b3c None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:16.303635 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35e812c2-7631-4c77-a986-04dbbf1c2b3c None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:16.303999 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35e812c2-7631-4c77-a986-04dbbf1c2b3c None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:16.324399 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:43:16.374628 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35e812c2-7631-4c77-a986-04dbbf1c2b3c None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:16.376004 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35e812c2-7631-4c77-a986-04dbbf1c2b3c None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:16.414263 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d92f9ea-26f9-4f3f-ab51-651890e9082a None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:16.431595 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:43:16.449756 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39eb9668-d5fd-46bb-bd3b-0c9bf58b57ec None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:16.450388 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39eb9668-d5fd-46bb-bd3b-0c9bf58b57ec None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:16.450670 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39eb9668-d5fd-46bb-bd3b-0c9bf58b57ec None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:16.450944 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39eb9668-d5fd-46bb-bd3b-0c9bf58b57ec None admin] PATH_INFO: `/v3/users/cinder` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:16.458098 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39eb9668-d5fd-46bb-bd3b-0c9bf58b57ec None admin] RBAC: Authorizing `identity:get_user(user_id=cinder)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:16.459537 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39eb9668-d5fd-46bb-bd3b-0c9bf58b57ec None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:16.471126 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application [None req-39eb9668-d5fd-46bb-bd3b-0c9bf58b57ec None admin] Could not find user: cinder.: keystone.exception.UserNotFound: Could not find user: cinder. Aug 30 01:43:16.471126 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 01:43:16.471126 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 01:43:16.471126 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 01:43:16.471126 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 01:43:16.471126 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 01:43:16.471126 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 01:43:16.471126 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 01:43:16.471126 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 01:43:16.471126 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 01:43:16.471126 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 01:43:16.471126 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 01:43:16.471126 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 30 01:43:16.471126 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 30 01:43:16.471126 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 30 01:43:16.471126 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 30 01:43:16.472010 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 01:43:16.472010 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 01:43:16.472010 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 30 01:43:16.472010 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 30 01:43:16.472010 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 30 01:43:16.472010 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 30 01:43:16.472010 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 01:43:16.472010 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 01:43:16.472010 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 01:43:16.472010 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 01:43:16.472010 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 01:43:16.472010 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application with Lock( Aug 30 01:43:16.472010 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 01:43:16.472010 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._enter() Aug 30 01:43:16.472010 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 01:43:16.472010 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 01:43:16.472010 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 01:43:16.472010 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.creator() Aug 30 01:43:16.472888 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 01:43:16.472888 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application created_value = creator( Aug 30 01:43:16.472888 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 30 01:43:16.472888 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 30 01:43:16.472888 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 30 01:43:16.472888 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 30 01:43:16.472888 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 30 01:43:16.472888 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 30 01:43:16.472888 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: cinder. Aug 30 01:43:16.472888 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Aug 30 01:43:16.472888 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 253/510] 10.222.0.27 () {56 vars in 1099 bytes} [Tue Aug 30 01:43:16 2022] GET /identity/v3/users/cinder => generated 84 bytes in 178 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 01:43:16.473498 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39eb9668-d5fd-46bb-bd3b-0c9bf58b57ec None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:16.484606 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 258/511] 10.222.0.27 () {58 vars in 1282 bytes} [Tue Aug 30 01:43:16 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 334 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 30 01:43:16.485530 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35e812c2-7631-4c77-a986-04dbbf1c2b3c None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:16.559447 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:43:16.573716 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39eb9668-d5fd-46bb-bd3b-0c9bf58b57ec None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:16.581464 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35e812c2-7631-4c77-a986-04dbbf1c2b3c None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:16.608427 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db84cde8-0043-48e4-807e-c1e4989ee193 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:16.609060 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db84cde8-0043-48e4-807e-c1e4989ee193 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:16.609341 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db84cde8-0043-48e4-807e-c1e4989ee193 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:16.609651 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db84cde8-0043-48e4-807e-c1e4989ee193 None admin] PATH_INFO: `/v3/domains` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:16.610285 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db84cde8-0043-48e4-807e-c1e4989ee193 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:43:16.610588 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db84cde8-0043-48e4-807e-c1e4989ee193 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:16.611982 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db84cde8-0043-48e4-807e-c1e4989ee193 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:16.615191 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-636fe6d3-ce7f-4b74-a6b7-260d1910f12b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:16.615781 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-636fe6d3-ce7f-4b74-a6b7-260d1910f12b None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:16.616106 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-636fe6d3-ce7f-4b74-a6b7-260d1910f12b None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:16.616950 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-636fe6d3-ce7f-4b74-a6b7-260d1910f12b None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:16.625171 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 254/512] 10.222.0.27 () {56 vars in 1109 bytes} [Tue Aug 30 01:43:16 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:43:16.625909 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db84cde8-0043-48e4-807e-c1e4989ee193 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:16.641543 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:43:16.684646 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-636fe6d3-ce7f-4b74-a6b7-260d1910f12b None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:16.686140 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-636fe6d3-ce7f-4b74-a6b7-260d1910f12b None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:16.729701 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db84cde8-0043-48e4-807e-c1e4989ee193 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:16.772208 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f09ce773-770b-40ae-ab39-db89e8dd61b2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:16.773032 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f09ce773-770b-40ae-ab39-db89e8dd61b2 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:16.773314 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f09ce773-770b-40ae-ab39-db89e8dd61b2 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:16.773626 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f09ce773-770b-40ae-ab39-db89e8dd61b2 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:16.783974 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f09ce773-770b-40ae-ab39-db89e8dd61b2 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:16.785425 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f09ce773-770b-40ae-ab39-db89e8dd61b2 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:16.788848 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 259/513] 10.222.0.27 () {58 vars in 1282 bytes} [Tue Aug 30 01:43:16 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 304 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 30 01:43:16.789703 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-636fe6d3-ce7f-4b74-a6b7-260d1910f12b None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:16.791675 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:43:16.800256 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application [None req-f09ce773-770b-40ae-ab39-db89e8dd61b2 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 01:43:16.800256 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 01:43:16.800256 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 01:43:16.800256 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 01:43:16.800256 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 01:43:16.800256 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 01:43:16.800256 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 01:43:16.800256 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 01:43:16.800256 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 01:43:16.800256 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 01:43:16.800256 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 01:43:16.800256 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 01:43:16.800256 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 30 01:43:16.800256 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 30 01:43:16.800256 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 30 01:43:16.800256 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 30 01:43:16.807130 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 01:43:16.807130 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 01:43:16.807130 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 01:43:16.807130 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 01:43:16.807130 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 01:43:16.807130 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 01:43:16.807130 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 01:43:16.807130 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application with Lock( Aug 30 01:43:16.807130 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 01:43:16.807130 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._enter() Aug 30 01:43:16.807130 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 01:43:16.807130 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 01:43:16.807130 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 01:43:16.807130 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.creator() Aug 30 01:43:16.807130 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 01:43:16.807130 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application created_value = creator( Aug 30 01:43:16.807130 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 30 01:43:16.807969 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 30 01:43:16.807969 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 01:43:16.807969 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Aug 30 01:43:16.807969 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 255/514] 10.222.0.27 () {56 vars in 1108 bytes} [Tue Aug 30 01:43:16 2022] GET /identity/v3/domains/Default => generated 87 bytes in 176 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 01:43:16.807969 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f09ce773-770b-40ae-ab39-db89e8dd61b2 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:16.844633 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:43:16.891325 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-636fe6d3-ce7f-4b74-a6b7-260d1910f12b None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:16.904135 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f09ce773-770b-40ae-ab39-db89e8dd61b2 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:16.924089 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66420d3f-07cf-4c72-8716-0126009b8776 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:16.924761 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66420d3f-07cf-4c72-8716-0126009b8776 None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:16.925109 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66420d3f-07cf-4c72-8716-0126009b8776 None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:16.925444 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66420d3f-07cf-4c72-8716-0126009b8776 None admin] PATH_INFO: `/v3/users` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:16.926203 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66420d3f-07cf-4c72-8716-0126009b8776 None admin] RBAC: Adding query filter params (domain_id=default, name=cinder) {{(pid=81578) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:43:16.926502 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66420d3f-07cf-4c72-8716-0126009b8776 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:16.927931 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66420d3f-07cf-4c72-8716-0126009b8776 None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:16.938033 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b45b87c3-82b5-4d41-bde2-0d8a3f43c5e5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:16.938656 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b45b87c3-82b5-4d41-bde2-0d8a3f43c5e5 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:16.938939 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b45b87c3-82b5-4d41-bde2-0d8a3f43c5e5 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:16.939212 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b45b87c3-82b5-4d41-bde2-0d8a3f43c5e5 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:16.950358 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b45b87c3-82b5-4d41-bde2-0d8a3f43c5e5 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:16.951829 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b45b87c3-82b5-4d41-bde2-0d8a3f43c5e5 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:16.964143 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 260/515] 10.222.0.27 () {56 vars in 1137 bytes} [Tue Aug 30 01:43:16 2022] GET /identity/v3/users?domain_id=default&name=cinder => generated 372 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:43:16.964963 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66420d3f-07cf-4c72-8716-0126009b8776 None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:16.966792 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application [None req-b45b87c3-82b5-4d41-bde2-0d8a3f43c5e5 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 01:43:16.966792 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 01:43:16.966792 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 01:43:16.966792 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 01:43:16.966792 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 01:43:16.966792 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 01:43:16.966792 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 01:43:16.966792 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 01:43:16.966792 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 01:43:16.966792 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 01:43:16.966792 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 01:43:16.966792 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 01:43:16.966792 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 30 01:43:16.966792 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 30 01:43:16.966792 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 30 01:43:16.966792 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 30 01:43:16.967694 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 01:43:16.967694 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 01:43:16.967694 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 01:43:16.967694 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 01:43:16.967694 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 01:43:16.967694 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 01:43:16.967694 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 01:43:16.967694 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application with Lock( Aug 30 01:43:16.967694 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 01:43:16.967694 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._enter() Aug 30 01:43:16.967694 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 01:43:16.967694 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 01:43:16.967694 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 01:43:16.967694 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.creator() Aug 30 01:43:16.967694 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 01:43:16.967694 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application created_value = creator( Aug 30 01:43:16.967694 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 30 01:43:16.969102 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 30 01:43:16.969102 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 01:43:16.969102 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Aug 30 01:43:16.969102 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 256/516] 10.222.0.27 () {56 vars in 1108 bytes} [Tue Aug 30 01:43:16 2022] GET /identity/v3/domains/Default => generated 87 bytes in 166 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 01:43:16.969102 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b45b87c3-82b5-4d41-bde2-0d8a3f43c5e5 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:17.018111 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:43:17.067156 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b45b87c3-82b5-4d41-bde2-0d8a3f43c5e5 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:17.067332 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66420d3f-07cf-4c72-8716-0126009b8776 None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:17.080350 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:43:17.099322 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f4b3785-fec7-48a3-9112-067274d254a9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:17.099881 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f4b3785-fec7-48a3-9112-067274d254a9 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:17.100157 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f4b3785-fec7-48a3-9112-067274d254a9 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:17.100258 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a32d69bd-8557-4619-a43d-405053d5e916 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:17.100949 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f4b3785-fec7-48a3-9112-067274d254a9 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:17.101740 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a32d69bd-8557-4619-a43d-405053d5e916 None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:17.102018 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a32d69bd-8557-4619-a43d-405053d5e916 None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:17.102283 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a32d69bd-8557-4619-a43d-405053d5e916 None admin] PATH_INFO: `/v3/users/neutron` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:17.109340 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a32d69bd-8557-4619-a43d-405053d5e916 None admin] RBAC: Authorizing `identity:get_user(user_id=neutron)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:17.110922 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a32d69bd-8557-4619-a43d-405053d5e916 None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:17.112053 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f4b3785-fec7-48a3-9112-067274d254a9 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:17.113570 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f4b3785-fec7-48a3-9112-067274d254a9 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:17.123817 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application [None req-a32d69bd-8557-4619-a43d-405053d5e916 None admin] Could not find user: neutron.: keystone.exception.UserNotFound: Could not find user: neutron. Aug 30 01:43:17.123817 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 01:43:17.123817 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 01:43:17.123817 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 01:43:17.123817 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 01:43:17.123817 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 01:43:17.123817 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 01:43:17.123817 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 01:43:17.123817 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 01:43:17.123817 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 01:43:17.123817 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 01:43:17.123817 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 01:43:17.123817 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 30 01:43:17.123817 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 30 01:43:17.123817 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 30 01:43:17.123817 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 30 01:43:17.124859 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 01:43:17.124859 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 01:43:17.124859 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 30 01:43:17.124859 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 30 01:43:17.124859 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 30 01:43:17.124859 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 30 01:43:17.124859 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 01:43:17.124859 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 01:43:17.124859 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 01:43:17.124859 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 01:43:17.124859 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 01:43:17.124859 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application with Lock( Aug 30 01:43:17.124859 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 01:43:17.124859 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._enter() Aug 30 01:43:17.124859 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 01:43:17.124859 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 01:43:17.124859 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 01:43:17.124859 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.creator() Aug 30 01:43:17.125815 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 01:43:17.125815 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application created_value = creator( Aug 30 01:43:17.125815 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 30 01:43:17.125815 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 30 01:43:17.125815 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 30 01:43:17.125815 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 30 01:43:17.125815 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 30 01:43:17.125815 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 30 01:43:17.125815 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: neutron. Aug 30 01:43:17.125815 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application Aug 30 01:43:17.125815 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 261/517] 10.222.0.27 () {56 vars in 1102 bytes} [Tue Aug 30 01:43:16 2022] GET /identity/v3/users/neutron => generated 85 bytes in 160 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 01:43:17.125815 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a32d69bd-8557-4619-a43d-405053d5e916 None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:17.127926 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application [None req-8f4b3785-fec7-48a3-9112-067274d254a9 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 01:43:17.127926 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 01:43:17.127926 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 01:43:17.127926 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 01:43:17.127926 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 01:43:17.127926 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 01:43:17.127926 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 01:43:17.127926 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 01:43:17.127926 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 01:43:17.127926 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 01:43:17.127926 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 01:43:17.127926 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 01:43:17.127926 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 30 01:43:17.127926 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 30 01:43:17.127926 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 30 01:43:17.127926 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 30 01:43:17.128812 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 01:43:17.128812 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 01:43:17.128812 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 01:43:17.128812 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 01:43:17.128812 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 01:43:17.128812 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 01:43:17.128812 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 01:43:17.128812 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application with Lock( Aug 30 01:43:17.128812 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 01:43:17.128812 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._enter() Aug 30 01:43:17.128812 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 01:43:17.128812 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 01:43:17.128812 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 01:43:17.128812 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.creator() Aug 30 01:43:17.128812 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 01:43:17.128812 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application created_value = creator( Aug 30 01:43:17.128812 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 30 01:43:17.130629 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 30 01:43:17.130629 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 01:43:17.130629 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Aug 30 01:43:17.130629 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 257/518] 10.222.0.27 () {56 vars in 1108 bytes} [Tue Aug 30 01:43:16 2022] GET /identity/v3/domains/Default => generated 87 bytes in 160 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 01:43:17.130629 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f4b3785-fec7-48a3-9112-067274d254a9 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:17.227808 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a32d69bd-8557-4619-a43d-405053d5e916 None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:17.229100 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f4b3785-fec7-48a3-9112-067274d254a9 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:17.230643 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:43:17.264185 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c693cd7c-4fb6-4e69-b520-b1357338feac None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:17.264479 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c621e79-2947-4fea-b3df-441fed628e43 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:17.264785 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c693cd7c-4fb6-4e69-b520-b1357338feac None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:17.265062 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c693cd7c-4fb6-4e69-b520-b1357338feac None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:17.265152 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c621e79-2947-4fea-b3df-441fed628e43 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:17.265330 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c693cd7c-4fb6-4e69-b520-b1357338feac None admin] PATH_INFO: `/v3/domains` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:17.265418 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c621e79-2947-4fea-b3df-441fed628e43 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:17.265633 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c621e79-2947-4fea-b3df-441fed628e43 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:17.265998 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c693cd7c-4fb6-4e69-b520-b1357338feac None admin] RBAC: Adding query filter params (name=Default) {{(pid=81578) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:43:17.266297 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c693cd7c-4fb6-4e69-b520-b1357338feac None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:17.267671 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c693cd7c-4fb6-4e69-b520-b1357338feac None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:17.279545 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 262/519] 10.222.0.27 () {56 vars in 1109 bytes} [Tue Aug 30 01:43:17 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:43:17.280353 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c693cd7c-4fb6-4e69-b520-b1357338feac None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:17.312833 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:43:17.334125 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c621e79-2947-4fea-b3df-441fed628e43 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:17.336005 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c621e79-2947-4fea-b3df-441fed628e43 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:17.380862 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c693cd7c-4fb6-4e69-b520-b1357338feac None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:17.413414 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b22c3fb0-1f25-4701-9ffb-9c15a79e452a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:17.414059 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b22c3fb0-1f25-4701-9ffb-9c15a79e452a None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:17.414324 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b22c3fb0-1f25-4701-9ffb-9c15a79e452a None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:17.414591 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b22c3fb0-1f25-4701-9ffb-9c15a79e452a None admin] PATH_INFO: `/v3/domains` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:17.415221 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b22c3fb0-1f25-4701-9ffb-9c15a79e452a None admin] RBAC: Adding query filter params (name=Default) {{(pid=81578) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:43:17.415518 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b22c3fb0-1f25-4701-9ffb-9c15a79e452a None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:17.416868 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b22c3fb0-1f25-4701-9ffb-9c15a79e452a None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:17.427631 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 263/520] 10.222.0.27 () {56 vars in 1109 bytes} [Tue Aug 30 01:43:17 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:43:17.428408 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b22c3fb0-1f25-4701-9ffb-9c15a79e452a None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:17.434952 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 258/521] 10.222.0.27 () {58 vars in 1282 bytes} [Tue Aug 30 01:43:17 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 305 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 30 01:43:17.435838 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c621e79-2947-4fea-b3df-441fed628e43 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:17.438117 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:43:17.524277 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:43:17.532496 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b22c3fb0-1f25-4701-9ffb-9c15a79e452a None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:17.535624 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c621e79-2947-4fea-b3df-441fed628e43 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:17.566345 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a61d2e29-f9e5-4fba-9d99-a4b68f943a69 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:17.566946 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a61d2e29-f9e5-4fba-9d99-a4b68f943a69 None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:17.567225 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a61d2e29-f9e5-4fba-9d99-a4b68f943a69 None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:17.567493 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a61d2e29-f9e5-4fba-9d99-a4b68f943a69 None admin] PATH_INFO: `/v3/users` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:17.568214 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a61d2e29-f9e5-4fba-9d99-a4b68f943a69 None admin] RBAC: Adding query filter params (domain_id=default, name=neutron) {{(pid=81578) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:43:17.568520 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a61d2e29-f9e5-4fba-9d99-a4b68f943a69 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:17.570025 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a61d2e29-f9e5-4fba-9d99-a4b68f943a69 None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:17.570118 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48824697-ba93-4f4e-9255-4cff6f132b00 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:17.570683 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48824697-ba93-4f4e-9255-4cff6f132b00 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:17.570958 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48824697-ba93-4f4e-9255-4cff6f132b00 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:17.571230 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48824697-ba93-4f4e-9255-4cff6f132b00 None admin] PATH_INFO: `/v3/domains` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:17.571856 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48824697-ba93-4f4e-9255-4cff6f132b00 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:43:17.572153 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48824697-ba93-4f4e-9255-4cff6f132b00 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:17.573524 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48824697-ba93-4f4e-9255-4cff6f132b00 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:17.584576 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 259/522] 10.222.0.27 () {56 vars in 1109 bytes} [Tue Aug 30 01:43:17 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:43:17.586296 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48824697-ba93-4f4e-9255-4cff6f132b00 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:17.587620 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 264/523] 10.222.0.27 () {56 vars in 1139 bytes} [Tue Aug 30 01:43:17 2022] GET /identity/v3/users?domain_id=default&name=neutron => generated 374 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:43:17.588592 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a61d2e29-f9e5-4fba-9d99-a4b68f943a69 None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:17.648294 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:43:17.685893 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48824697-ba93-4f4e-9255-4cff6f132b00 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:17.686148 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a61d2e29-f9e5-4fba-9d99-a4b68f943a69 None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:17.720826 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3625c68-b355-4d3a-8e90-271d68ec8de6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:17.721631 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3625c68-b355-4d3a-8e90-271d68ec8de6 None admin] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:17.721723 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3625c68-b355-4d3a-8e90-271d68ec8de6 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:17.721989 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3625c68-b355-4d3a-8e90-271d68ec8de6 None admin] PATH_INFO: `/v3/users` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:17.722649 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3625c68-b355-4d3a-8e90-271d68ec8de6 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:17.724067 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3625c68-b355-4d3a-8e90-271d68ec8de6 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:17.724176 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c247f973-51ca-4030-8b6e-fb4e10b0ab7c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:17.724769 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c247f973-51ca-4030-8b6e-fb4e10b0ab7c None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:17.725044 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c247f973-51ca-4030-8b6e-fb4e10b0ab7c None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:17.725312 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c247f973-51ca-4030-8b6e-fb4e10b0ab7c None admin] PATH_INFO: `/v3/projects/service` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:17.733031 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c247f973-51ca-4030-8b6e-fb4e10b0ab7c None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:17.734359 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c247f973-51ca-4030-8b6e-fb4e10b0ab7c None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:17.746212 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application [None req-c247f973-51ca-4030-8b6e-fb4e10b0ab7c None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 30 01:43:17.746212 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 01:43:17.746212 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 01:43:17.746212 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 01:43:17.746212 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 01:43:17.746212 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 01:43:17.746212 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 01:43:17.746212 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 01:43:17.746212 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 01:43:17.746212 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 01:43:17.746212 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 01:43:17.746212 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 01:43:17.746212 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 30 01:43:17.746212 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 30 01:43:17.746212 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 30 01:43:17.747159 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 30 01:43:17.747159 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 01:43:17.747159 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 01:43:17.747159 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 01:43:17.747159 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 01:43:17.747159 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 01:43:17.747159 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 01:43:17.747159 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 01:43:17.747159 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application with Lock( Aug 30 01:43:17.747159 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 01:43:17.747159 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._enter() Aug 30 01:43:17.747159 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 01:43:17.747159 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 01:43:17.747159 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 01:43:17.747159 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.creator() Aug 30 01:43:17.747159 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 01:43:17.747159 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application created_value = creator( Aug 30 01:43:17.748009 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 30 01:43:17.748009 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 30 01:43:17.748009 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 30 01:43:17.748009 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 30 01:43:17.748009 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 30 01:43:17.748009 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 30 01:43:17.748009 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 30 01:43:17.748009 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application Aug 30 01:43:17.748009 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 265/524] 10.222.0.27 () {56 vars in 1111 bytes} [Tue Aug 30 01:43:17 2022] GET /identity/v3/projects/service => generated 88 bytes in 159 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 01:43:17.748489 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c247f973-51ca-4030-8b6e-fb4e10b0ab7c None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:17.764521 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:43:17.801760 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 260/525] 10.222.0.27 () {60 vars in 1132 bytes} [Tue Aug 30 01:43:17 2022] POST /identity/v3/users => generated 247 bytes in 217 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 01:43:17.802588 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3625c68-b355-4d3a-8e90-271d68ec8de6 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:17.857994 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c247f973-51ca-4030-8b6e-fb4e10b0ab7c None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:17.898846 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22b3571c-10da-4d05-a564-6d9afac83440 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:17.899511 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22b3571c-10da-4d05-a564-6d9afac83440 None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:17.899673 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22b3571c-10da-4d05-a564-6d9afac83440 None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:17.899944 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22b3571c-10da-4d05-a564-6d9afac83440 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:17.911500 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22b3571c-10da-4d05-a564-6d9afac83440 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:17.913016 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22b3571c-10da-4d05-a564-6d9afac83440 None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:17.926331 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3625c68-b355-4d3a-8e90-271d68ec8de6 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:17.928687 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application [None req-22b3571c-10da-4d05-a564-6d9afac83440 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 01:43:17.928687 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 01:43:17.928687 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 01:43:17.928687 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 01:43:17.928687 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 01:43:17.928687 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 01:43:17.928687 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 01:43:17.928687 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 01:43:17.928687 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 01:43:17.928687 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 01:43:17.928687 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 01:43:17.928687 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 01:43:17.928687 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 30 01:43:17.928687 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 30 01:43:17.928687 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 30 01:43:17.928687 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 30 01:43:17.929879 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 01:43:17.929879 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 01:43:17.929879 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 01:43:17.929879 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 01:43:17.929879 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 01:43:17.929879 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 01:43:17.929879 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 01:43:17.929879 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application with Lock( Aug 30 01:43:17.929879 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 01:43:17.929879 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._enter() Aug 30 01:43:17.929879 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 01:43:17.929879 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 01:43:17.929879 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 01:43:17.929879 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.creator() Aug 30 01:43:17.929879 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 01:43:17.929879 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application created_value = creator( Aug 30 01:43:17.929879 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 30 01:43:17.930761 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 30 01:43:17.930761 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 01:43:17.930761 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application Aug 30 01:43:17.930761 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 266/526] 10.222.0.27 () {56 vars in 1108 bytes} [Tue Aug 30 01:43:17 2022] GET /identity/v3/domains/Default => generated 87 bytes in 182 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 01:43:17.930761 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22b3571c-10da-4d05-a564-6d9afac83440 None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:17.952433 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:43:17.971596 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed3e2c16-0cfe-4887-84c1-45de9cb2ddf9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:17.972355 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed3e2c16-0cfe-4887-84c1-45de9cb2ddf9 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:17.973578 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed3e2c16-0cfe-4887-84c1-45de9cb2ddf9 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:17.973695 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed3e2c16-0cfe-4887-84c1-45de9cb2ddf9 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:17.982718 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed3e2c16-0cfe-4887-84c1-45de9cb2ddf9 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:17.985060 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed3e2c16-0cfe-4887-84c1-45de9cb2ddf9 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:18.000500 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application [None req-ed3e2c16-0cfe-4887-84c1-45de9cb2ddf9 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 30 01:43:18.000500 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 01:43:18.000500 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 01:43:18.000500 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 01:43:18.000500 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 01:43:18.000500 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 01:43:18.000500 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 01:43:18.000500 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 01:43:18.000500 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 01:43:18.000500 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 01:43:18.000500 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 01:43:18.000500 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 01:43:18.000500 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 30 01:43:18.000500 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 30 01:43:18.000500 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 30 01:43:18.001398 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 30 01:43:18.001398 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 01:43:18.001398 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 01:43:18.001398 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 01:43:18.001398 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 01:43:18.001398 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 01:43:18.001398 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 01:43:18.001398 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 01:43:18.001398 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application with Lock( Aug 30 01:43:18.001398 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 01:43:18.001398 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._enter() Aug 30 01:43:18.001398 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 01:43:18.001398 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 01:43:18.001398 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 01:43:18.001398 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.creator() Aug 30 01:43:18.001398 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 01:43:18.001398 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application created_value = creator( Aug 30 01:43:18.002361 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 30 01:43:18.002361 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 30 01:43:18.002361 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 30 01:43:18.002361 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 30 01:43:18.002361 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 30 01:43:18.002361 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 30 01:43:18.002361 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 30 01:43:18.002361 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Aug 30 01:43:18.002361 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 261/527] 10.222.0.27 () {56 vars in 1111 bytes} [Tue Aug 30 01:43:17 2022] GET /identity/v3/projects/service => generated 88 bytes in 199 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 01:43:18.002860 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed3e2c16-0cfe-4887-84c1-45de9cb2ddf9 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:18.007902 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:43:18.049730 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22b3571c-10da-4d05-a564-6d9afac83440 None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:18.088672 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-957c6fdb-2442-4853-9abc-e773c6dbf4a1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:18.089333 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-957c6fdb-2442-4853-9abc-e773c6dbf4a1 None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:18.089642 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-957c6fdb-2442-4853-9abc-e773c6dbf4a1 None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:18.089937 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-957c6fdb-2442-4853-9abc-e773c6dbf4a1 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:18.122270 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed3e2c16-0cfe-4887-84c1-45de9cb2ddf9 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:18.156919 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be1fec18-37b0-4b54-a9e7-b770ec86e7b0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:18.157604 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be1fec18-37b0-4b54-a9e7-b770ec86e7b0 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:18.157915 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be1fec18-37b0-4b54-a9e7-b770ec86e7b0 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:18.158187 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be1fec18-37b0-4b54-a9e7-b770ec86e7b0 None admin] PATH_INFO: `/v3/projects` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:18.158881 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be1fec18-37b0-4b54-a9e7-b770ec86e7b0 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:43:18.159189 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be1fec18-37b0-4b54-a9e7-b770ec86e7b0 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:18.160620 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be1fec18-37b0-4b54-a9e7-b770ec86e7b0 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:18.169386 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-957c6fdb-2442-4853-9abc-e773c6dbf4a1 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:18.170840 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-957c6fdb-2442-4853-9abc-e773c6dbf4a1 None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:18.184254 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 262/528] 10.222.0.27 () {56 vars in 1148 bytes} [Tue Aug 30 01:43:18 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:43:18.184254 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be1fec18-37b0-4b54-a9e7-b770ec86e7b0 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:18.224042 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:43:18.242864 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:43:18.286931 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 267/529] 10.222.0.27 () {58 vars in 1282 bytes} [Tue Aug 30 01:43:17 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 355 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 30 01:43:18.286931 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-957c6fdb-2442-4853-9abc-e773c6dbf4a1 None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:18.290866 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be1fec18-37b0-4b54-a9e7-b770ec86e7b0 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:18.327225 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e1c7836-7ac5-4864-a519-211c75e98927 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:18.328206 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e1c7836-7ac5-4864-a519-211c75e98927 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:18.328778 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e1c7836-7ac5-4864-a519-211c75e98927 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:18.329189 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e1c7836-7ac5-4864-a519-211c75e98927 None admin] PATH_INFO: `/v3/domains` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:18.330004 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e1c7836-7ac5-4864-a519-211c75e98927 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:43:18.330503 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e1c7836-7ac5-4864-a519-211c75e98927 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:18.332037 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e1c7836-7ac5-4864-a519-211c75e98927 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:18.344894 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 263/530] 10.222.0.27 () {56 vars in 1109 bytes} [Tue Aug 30 01:43:18 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:43:18.345907 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e1c7836-7ac5-4864-a519-211c75e98927 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:18.392363 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-957c6fdb-2442-4853-9abc-e773c6dbf4a1 None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:18.427926 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e7b538c-f034-465e-b7d6-fda6e90d513e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:18.429284 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e7b538c-f034-465e-b7d6-fda6e90d513e None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:18.429600 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e7b538c-f034-465e-b7d6-fda6e90d513e None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:18.429961 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e7b538c-f034-465e-b7d6-fda6e90d513e None admin] PATH_INFO: `/v3/projects` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:18.430654 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e7b538c-f034-465e-b7d6-fda6e90d513e None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81578) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:43:18.430960 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e7b538c-f034-465e-b7d6-fda6e90d513e None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:18.432382 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e7b538c-f034-465e-b7d6-fda6e90d513e None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:18.450870 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 268/531] 10.222.0.27 () {56 vars in 1148 bytes} [Tue Aug 30 01:43:18 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:43:18.451078 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:43:18.452028 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e7b538c-f034-465e-b7d6-fda6e90d513e None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:18.459466 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e1c7836-7ac5-4864-a519-211c75e98927 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:18.477576 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:43:18.494672 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7518aab3-3372-4ee3-949e-edb6e625dd00 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:18.495281 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7518aab3-3372-4ee3-949e-edb6e625dd00 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:18.495561 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7518aab3-3372-4ee3-949e-edb6e625dd00 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:18.495843 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7518aab3-3372-4ee3-949e-edb6e625dd00 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:18.496571 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7518aab3-3372-4ee3-949e-edb6e625dd00 None admin] RBAC: Adding query filter params (role.id=aa59b0c9009a480994d9df81a5f8d404, scope.project.id=4562f3951fbe465280073e60a11e44e2, user.id=3caff62093d748f48596574836e12e42) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:43:18.496879 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7518aab3-3372-4ee3-949e-edb6e625dd00 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:18.498385 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7518aab3-3372-4ee3-949e-edb6e625dd00 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:18.503713 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 264/532] 10.222.0.27 () {56 vars in 1374 bytes} [Tue Aug 30 01:43:18 2022] GET /identity/v3/role_assignments?user.id=3caff62093d748f48596574836e12e42&scope.project.id=4562f3951fbe465280073e60a11e44e2&role.id=aa59b0c9009a480994d9df81a5f8d404 => generated 259 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:43:18.505946 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7518aab3-3372-4ee3-949e-edb6e625dd00 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:18.575228 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e7b538c-f034-465e-b7d6-fda6e90d513e None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:18.624584 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab0c5c11-4f7b-45fe-98bf-88719c94d35f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:18.625837 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab0c5c11-4f7b-45fe-98bf-88719c94d35f None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:18.626517 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab0c5c11-4f7b-45fe-98bf-88719c94d35f None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:18.626928 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab0c5c11-4f7b-45fe-98bf-88719c94d35f None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:18.628363 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7518aab3-3372-4ee3-949e-edb6e625dd00 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:18.640860 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab0c5c11-4f7b-45fe-98bf-88719c94d35f None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:18.643127 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab0c5c11-4f7b-45fe-98bf-88719c94d35f None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:18.663196 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application [None req-ab0c5c11-4f7b-45fe-98bf-88719c94d35f None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 01:43:18.663196 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 01:43:18.663196 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 01:43:18.663196 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 01:43:18.663196 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 01:43:18.663196 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 01:43:18.663196 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 01:43:18.663196 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 01:43:18.663196 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 01:43:18.663196 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 01:43:18.663196 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 01:43:18.663196 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 01:43:18.663196 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 30 01:43:18.663196 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 30 01:43:18.663196 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 30 01:43:18.663196 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 30 01:43:18.664320 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 01:43:18.664320 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 01:43:18.664320 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 01:43:18.664320 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 01:43:18.664320 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 01:43:18.664320 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 01:43:18.664320 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 01:43:18.664320 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application with Lock( Aug 30 01:43:18.664320 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 01:43:18.664320 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._enter() Aug 30 01:43:18.664320 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 01:43:18.664320 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 01:43:18.664320 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 01:43:18.664320 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.creator() Aug 30 01:43:18.664320 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 01:43:18.664320 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application created_value = creator( Aug 30 01:43:18.664320 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 30 01:43:18.665152 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 30 01:43:18.665152 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 01:43:18.665152 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application Aug 30 01:43:18.667113 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 269/533] 10.222.0.27 () {56 vars in 1108 bytes} [Tue Aug 30 01:43:18 2022] GET /identity/v3/domains/Default => generated 87 bytes in 215 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 01:43:18.668112 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab0c5c11-4f7b-45fe-98bf-88719c94d35f None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:18.679546 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e0dc174-440f-4154-8007-e475ed51fdfd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:18.680114 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e0dc174-440f-4154-8007-e475ed51fdfd None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:18.680393 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e0dc174-440f-4154-8007-e475ed51fdfd None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:18.680684 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e0dc174-440f-4154-8007-e475ed51fdfd None admin] PATH_INFO: `/v3/projects/service` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:18.693103 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e0dc174-440f-4154-8007-e475ed51fdfd None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:18.695637 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e0dc174-440f-4154-8007-e475ed51fdfd None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:18.712755 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application [None req-4e0dc174-440f-4154-8007-e475ed51fdfd None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 30 01:43:18.712755 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 01:43:18.712755 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 01:43:18.712755 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 01:43:18.712755 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 01:43:18.712755 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 01:43:18.712755 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 01:43:18.712755 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 01:43:18.712755 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 01:43:18.712755 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 01:43:18.712755 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 01:43:18.712755 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 01:43:18.712755 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 30 01:43:18.712755 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 30 01:43:18.712755 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 30 01:43:18.713969 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 30 01:43:18.713969 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 01:43:18.713969 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 01:43:18.713969 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 01:43:18.713969 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 01:43:18.713969 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 01:43:18.713969 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 01:43:18.713969 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 01:43:18.713969 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application with Lock( Aug 30 01:43:18.713969 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 01:43:18.713969 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._enter() Aug 30 01:43:18.713969 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 01:43:18.713969 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 01:43:18.713969 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 01:43:18.713969 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.creator() Aug 30 01:43:18.713969 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 01:43:18.713969 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application created_value = creator( Aug 30 01:43:18.714816 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 30 01:43:18.714816 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 30 01:43:18.714816 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 30 01:43:18.714816 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 30 01:43:18.714816 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 30 01:43:18.714816 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 30 01:43:18.714816 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 30 01:43:18.714816 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Aug 30 01:43:18.714816 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 265/534] 10.222.0.27 () {56 vars in 1111 bytes} [Tue Aug 30 01:43:18 2022] GET /identity/v3/projects/service => generated 88 bytes in 210 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 01:43:18.715271 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e0dc174-440f-4154-8007-e475ed51fdfd None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:18.751349 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:43:18.774132 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:43:18.810696 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab0c5c11-4f7b-45fe-98bf-88719c94d35f None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:18.849114 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-534a3391-9690-45a9-8429-afa2552f390e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:18.850464 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-534a3391-9690-45a9-8429-afa2552f390e None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:18.851008 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-534a3391-9690-45a9-8429-afa2552f390e None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:18.851778 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-534a3391-9690-45a9-8429-afa2552f390e None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:18.852712 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-534a3391-9690-45a9-8429-afa2552f390e None admin] RBAC: Adding query filter params (role.id=aa59b0c9009a480994d9df81a5f8d404, scope.project.id=4562f3951fbe465280073e60a11e44e2, user.id=d64f2e71361644589bec8a06e6eceed1) {{(pid=81578) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:43:18.853216 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-534a3391-9690-45a9-8429-afa2552f390e None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:18.853660 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e0dc174-440f-4154-8007-e475ed51fdfd None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:18.855203 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-534a3391-9690-45a9-8429-afa2552f390e None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:18.861766 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 270/535] 10.222.0.27 () {56 vars in 1374 bytes} [Tue Aug 30 01:43:18 2022] GET /identity/v3/role_assignments?user.id=d64f2e71361644589bec8a06e6eceed1&scope.project.id=4562f3951fbe465280073e60a11e44e2&role.id=aa59b0c9009a480994d9df81a5f8d404 => generated 259 bytes in 194 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:43:18.863063 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-534a3391-9690-45a9-8429-afa2552f390e None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:18.903053 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92718d93-8764-4fc7-b97c-d4fe68f03f6f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:18.903685 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92718d93-8764-4fc7-b97c-d4fe68f03f6f None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:18.903965 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92718d93-8764-4fc7-b97c-d4fe68f03f6f None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:18.904240 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92718d93-8764-4fc7-b97c-d4fe68f03f6f None admin] PATH_INFO: `/v3/domains` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:18.904901 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92718d93-8764-4fc7-b97c-d4fe68f03f6f None admin] RBAC: Adding query filter params (name=Default) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:43:18.905206 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92718d93-8764-4fc7-b97c-d4fe68f03f6f None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:18.906965 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92718d93-8764-4fc7-b97c-d4fe68f03f6f None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:18.924059 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 266/536] 10.222.0.27 () {56 vars in 1109 bytes} [Tue Aug 30 01:43:18 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 209 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:43:18.930353 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92718d93-8764-4fc7-b97c-d4fe68f03f6f None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:18.986743 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-534a3391-9690-45a9-8429-afa2552f390e None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:19.023242 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:43:19.026957 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-743f48be-e6c8-4f5a-86a2-13d1c2c4168c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:19.027858 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-743f48be-e6c8-4f5a-86a2-13d1c2c4168c None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:19.028353 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-743f48be-e6c8-4f5a-86a2-13d1c2c4168c None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:19.028830 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-743f48be-e6c8-4f5a-86a2-13d1c2c4168c None admin] PATH_INFO: `/v3/projects` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:19.029775 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-743f48be-e6c8-4f5a-86a2-13d1c2c4168c None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81578) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:43:19.030294 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-743f48be-e6c8-4f5a-86a2-13d1c2c4168c None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:19.031909 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-743f48be-e6c8-4f5a-86a2-13d1c2c4168c None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:19.040273 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:43:19.043889 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92718d93-8764-4fc7-b97c-d4fe68f03f6f None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:19.047352 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 271/537] 10.222.0.27 () {56 vars in 1148 bytes} [Tue Aug 30 01:43:18 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:43:19.055038 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-743f48be-e6c8-4f5a-86a2-13d1c2c4168c None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:19.090373 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc6f7f23-d55e-4a94-a83d-c0b677abaa12 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:19.091061 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc6f7f23-d55e-4a94-a83d-c0b677abaa12 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:19.091361 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc6f7f23-d55e-4a94-a83d-c0b677abaa12 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:19.091618 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc6f7f23-d55e-4a94-a83d-c0b677abaa12 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:19.106670 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc6f7f23-d55e-4a94-a83d-c0b677abaa12 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:19.108140 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc6f7f23-d55e-4a94-a83d-c0b677abaa12 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:19.134788 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application [None req-fc6f7f23-d55e-4a94-a83d-c0b677abaa12 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 30 01:43:19.134788 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 01:43:19.134788 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 01:43:19.134788 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 01:43:19.134788 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 01:43:19.134788 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 01:43:19.134788 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 01:43:19.134788 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 01:43:19.134788 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 01:43:19.134788 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 01:43:19.134788 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 01:43:19.134788 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 01:43:19.134788 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 30 01:43:19.134788 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 30 01:43:19.134788 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 30 01:43:19.135766 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 30 01:43:19.135766 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 01:43:19.135766 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 01:43:19.135766 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 01:43:19.135766 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 01:43:19.135766 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 01:43:19.135766 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 01:43:19.135766 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 01:43:19.135766 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application with Lock( Aug 30 01:43:19.135766 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 01:43:19.135766 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._enter() Aug 30 01:43:19.135766 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 01:43:19.135766 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 01:43:19.135766 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 01:43:19.135766 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.creator() Aug 30 01:43:19.135766 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 01:43:19.135766 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application created_value = creator( Aug 30 01:43:19.136727 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 30 01:43:19.136727 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 30 01:43:19.136727 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 30 01:43:19.136727 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 30 01:43:19.136727 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 30 01:43:19.136727 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 30 01:43:19.136727 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 30 01:43:19.136727 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Aug 30 01:43:19.136727 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 267/538] 10.222.0.27 () {56 vars in 1111 bytes} [Tue Aug 30 01:43:18 2022] GET /identity/v3/projects/service => generated 88 bytes in 207 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 30 01:43:19.141402 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc6f7f23-d55e-4a94-a83d-c0b677abaa12 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:19.189336 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-743f48be-e6c8-4f5a-86a2-13d1c2c4168c None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:19.225349 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fac126ac-d039-4ad6-874f-28ba97666af1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:19.226118 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fac126ac-d039-4ad6-874f-28ba97666af1 None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:19.226710 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fac126ac-d039-4ad6-874f-28ba97666af1 None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:19.226982 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fac126ac-d039-4ad6-874f-28ba97666af1 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:19.227765 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fac126ac-d039-4ad6-874f-28ba97666af1 None admin] RBAC: Adding query filter params (role.id=aa59b0c9009a480994d9df81a5f8d404, scope.project.id=4562f3951fbe465280073e60a11e44e2, user.id=ab936b3767a8417aa80c8c8f1b84b667) {{(pid=81578) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:43:19.228073 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fac126ac-d039-4ad6-874f-28ba97666af1 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:19.229578 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fac126ac-d039-4ad6-874f-28ba97666af1 None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:19.237384 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 272/539] 10.222.0.27 () {56 vars in 1374 bytes} [Tue Aug 30 01:43:19 2022] GET /identity/v3/role_assignments?user.id=ab936b3767a8417aa80c8c8f1b84b667&scope.project.id=4562f3951fbe465280073e60a11e44e2&role.id=aa59b0c9009a480994d9df81a5f8d404 => generated 259 bytes in 183 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:43:19.281570 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc6f7f23-d55e-4a94-a83d-c0b677abaa12 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:19.322736 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d38d6333-ad74-46e7-b25a-956f475119fb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:19.323350 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d38d6333-ad74-46e7-b25a-956f475119fb None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:19.323623 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d38d6333-ad74-46e7-b25a-956f475119fb None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:19.323890 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d38d6333-ad74-46e7-b25a-956f475119fb None admin] PATH_INFO: `/v3/projects` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:19.324651 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d38d6333-ad74-46e7-b25a-956f475119fb None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:43:19.324957 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d38d6333-ad74-46e7-b25a-956f475119fb None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:19.326445 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d38d6333-ad74-46e7-b25a-956f475119fb None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:19.342077 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 268/540] 10.222.0.27 () {56 vars in 1148 bytes} [Tue Aug 30 01:43:19 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 201 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 30 01:43:19.347993 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fac126ac-d039-4ad6-874f-28ba97666af1 None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:19.431072 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:43:19.473771 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fac126ac-d039-4ad6-874f-28ba97666af1 None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:19.508968 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1befa73b-50f1-476d-ae88-c41b06e55c01 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:19.509601 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1befa73b-50f1-476d-ae88-c41b06e55c01 None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:19.509876 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1befa73b-50f1-476d-ae88-c41b06e55c01 None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:19.510196 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1befa73b-50f1-476d-ae88-c41b06e55c01 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:19.510925 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1befa73b-50f1-476d-ae88-c41b06e55c01 None admin] RBAC: Adding query filter params (role.id=aa59b0c9009a480994d9df81a5f8d404, scope.project.id=4562f3951fbe465280073e60a11e44e2, user.id=f3ea1580c56f4814a92bbb2acf542905) {{(pid=81578) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:43:19.511231 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1befa73b-50f1-476d-ae88-c41b06e55c01 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:19.512656 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1befa73b-50f1-476d-ae88-c41b06e55c01 None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:19.518461 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 273/541] 10.222.0.27 () {56 vars in 1374 bytes} [Tue Aug 30 01:43:19 2022] GET /identity/v3/role_assignments?user.id=f3ea1580c56f4814a92bbb2acf542905&scope.project.id=4562f3951fbe465280073e60a11e44e2&role.id=aa59b0c9009a480994d9df81a5f8d404 => generated 259 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:43:19.889214 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4bed38e-92e5-4671-bbc9-1dcbf46162e8 None None] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:19.889612 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4bed38e-92e5-4671-bbc9-1dcbf46162e8 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:19.889809 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4bed38e-92e5-4671-bbc9-1dcbf46162e8 None None] PATH_INFO: `/` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:19.891403 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 269/542] 10.222.0.27 () {54 vars in 927 bytes} [Tue Aug 30 01:43:19 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 01:43:19.896799 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9e708f1-d4df-4450-a6b2-a688fa0a3c6c None None] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:19.897090 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9e708f1-d4df-4450-a6b2-a688fa0a3c6c None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:19.898021 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9e708f1-d4df-4450-a6b2-a688fa0a3c6c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:19.926042 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.auth.core [None req-e9e708f1-d4df-4450-a6b2-a688fa0a3c6c None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81578) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 01:43:19.927276 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-e9e708f1-d4df-4450-a6b2-a688fa0a3c6c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:43:19.998018 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 274/543] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Aug 30 01:43:19 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 30 01:43:20.336775 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4bed38e-92e5-4671-bbc9-1dcbf46162e8 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:20.340276 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:43:20.478167 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4bed38e-92e5-4671-bbc9-1dcbf46162e8 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:20.530881 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfa57b1b-d444-4414-9ce6-de7142f01ecb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:20.531855 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfa57b1b-d444-4414-9ce6-de7142f01ecb None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:20.532359 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfa57b1b-d444-4414-9ce6-de7142f01ecb None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:20.532824 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfa57b1b-d444-4414-9ce6-de7142f01ecb None admin] PATH_INFO: `/v3/roles/service` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:20.549993 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dfa57b1b-d444-4414-9ce6-de7142f01ecb None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 30 01:43:20.556419 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application [None req-dfa57b1b-d444-4414-9ce6-de7142f01ecb None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Aug 30 01:43:20.556419 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 01:43:20.556419 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 01:43:20.556419 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 01:43:20.556419 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 01:43:20.556419 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 01:43:20.556419 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 01:43:20.556419 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 01:43:20.556419 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 01:43:20.556419 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 01:43:20.556419 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 01:43:20.556419 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 01:43:20.556419 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 30 01:43:20.556419 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 30 01:43:20.556419 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 30 01:43:20.556419 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 30 01:43:20.557298 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 30 01:43:20.557298 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 30 01:43:20.557298 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 30 01:43:20.557298 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 30 01:43:20.557298 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 01:43:20.557298 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 01:43:20.557298 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 01:43:20.557298 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 01:43:20.557298 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 01:43:20.557298 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 01:43:20.557298 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 01:43:20.557298 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application with Lock( Aug 30 01:43:20.557298 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 01:43:20.557298 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._enter() Aug 30 01:43:20.557298 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 01:43:20.557298 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 01:43:20.558149 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 01:43:20.558149 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.creator() Aug 30 01:43:20.558149 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 01:43:20.558149 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application created_value = creator( Aug 30 01:43:20.558149 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 30 01:43:20.558149 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 30 01:43:20.558149 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 30 01:43:20.558149 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 30 01:43:20.558149 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 30 01:43:20.558149 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 30 01:43:20.558149 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Aug 30 01:43:20.558149 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Aug 30 01:43:20.566419 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 270/544] 10.222.0.27 () {56 vars in 1102 bytes} [Tue Aug 30 01:43:20 2022] GET /identity/v3/roles/service => generated 85 bytes in 230 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 01:43:20.574692 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9e708f1-d4df-4450-a6b2-a688fa0a3c6c None None] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:20.657169 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:43:20.708145 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9e708f1-d4df-4450-a6b2-a688fa0a3c6c None None] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:20.763112 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6898ef7b-9a34-4b10-8d28-586b2e482113 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:20.763732 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6898ef7b-9a34-4b10-8d28-586b2e482113 None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:20.764053 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6898ef7b-9a34-4b10-8d28-586b2e482113 None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:20.764323 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6898ef7b-9a34-4b10-8d28-586b2e482113 None admin] PATH_INFO: `/v3/roles` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:20.764987 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6898ef7b-9a34-4b10-8d28-586b2e482113 None admin] RBAC: Adding query filter params (name=service) {{(pid=81578) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:43:20.765284 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6898ef7b-9a34-4b10-8d28-586b2e482113 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:20.766667 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6898ef7b-9a34-4b10-8d28-586b2e482113 None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:20.782917 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 275/545] 10.222.0.27 () {56 vars in 1103 bytes} [Tue Aug 30 01:43:20 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 208 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:43:20.789313 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfa57b1b-d444-4414-9ce6-de7142f01ecb None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:20.901379 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:43:20.948765 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfa57b1b-d444-4414-9ce6-de7142f01ecb None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:20.984997 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b8ec431-84ef-4350-9770-bd8a546ff100 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:20.985673 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b8ec431-84ef-4350-9770-bd8a546ff100 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:20.985955 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b8ec431-84ef-4350-9770-bd8a546ff100 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:20.986229 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b8ec431-84ef-4350-9770-bd8a546ff100 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:20.996673 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c65acff-44d7-4e49-87e2-9045766594b4 None None] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:20.996955 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c65acff-44d7-4e49-87e2-9045766594b4 None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:20.997429 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c65acff-44d7-4e49-87e2-9045766594b4 None None] PATH_INFO: `/` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:20.999081 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 276/546] 10.222.0.27 () {54 vars in 927 bytes} [Tue Aug 30 01:43:20 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 01:43:21.005999 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b8ec431-84ef-4350-9770-bd8a546ff100 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:21.007885 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2441e9b7-b88e-4116-85a4-2b59182e154e None None] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:21.008170 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2441e9b7-b88e-4116-85a4-2b59182e154e None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:21.008437 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2441e9b7-b88e-4116-85a4-2b59182e154e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:21.009647 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b8ec431-84ef-4350-9770-bd8a546ff100 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:21.028218 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application [None req-5b8ec431-84ef-4350-9770-bd8a546ff100 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 01:43:21.028218 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 01:43:21.028218 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 01:43:21.028218 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 01:43:21.028218 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 01:43:21.028218 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 01:43:21.028218 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 01:43:21.028218 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 01:43:21.028218 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 01:43:21.028218 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 01:43:21.028218 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 01:43:21.028218 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 01:43:21.028218 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 30 01:43:21.028218 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 30 01:43:21.028218 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 30 01:43:21.028218 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 30 01:43:21.029248 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 01:43:21.029248 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 01:43:21.029248 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 01:43:21.029248 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 01:43:21.029248 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 01:43:21.029248 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 01:43:21.029248 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 01:43:21.029248 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application with Lock( Aug 30 01:43:21.029248 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 01:43:21.029248 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._enter() Aug 30 01:43:21.029248 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 01:43:21.029248 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 01:43:21.029248 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 01:43:21.029248 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.creator() Aug 30 01:43:21.029248 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 01:43:21.029248 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application created_value = creator( Aug 30 01:43:21.029248 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 30 01:43:21.030485 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 30 01:43:21.030485 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 01:43:21.030485 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Aug 30 01:43:21.032464 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 271/547] 10.222.0.27 () {56 vars in 1108 bytes} [Tue Aug 30 01:43:20 2022] GET /identity/v3/domains/Default => generated 87 bytes in 244 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 30 01:43:21.038413 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b8ec431-84ef-4350-9770-bd8a546ff100 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:21.049128 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.auth.core [None req-2441e9b7-b88e-4116-85a4-2b59182e154e None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81578) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 01:43:21.050500 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-2441e9b7-b88e-4116-85a4-2b59182e154e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:43:21.153114 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b8ec431-84ef-4350-9770-bd8a546ff100 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:21.185837 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:43:21.192972 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 277/548] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Aug 30 01:43:21 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 185 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Aug 30 01:43:21.203821 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9310cbd-4aec-4246-9335-0c94724b34ca None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:21.204362 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9310cbd-4aec-4246-9335-0c94724b34ca None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:21.204643 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9310cbd-4aec-4246-9335-0c94724b34ca None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:21.204915 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9310cbd-4aec-4246-9335-0c94724b34ca None admin] PATH_INFO: `/v3/domains` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:21.205608 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9310cbd-4aec-4246-9335-0c94724b34ca None admin] RBAC: Adding query filter params (name=Default) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:43:21.205900 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9310cbd-4aec-4246-9335-0c94724b34ca None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:21.207378 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9310cbd-4aec-4246-9335-0c94724b34ca None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:21.221190 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 272/549] 10.222.0.27 () {56 vars in 1109 bytes} [Tue Aug 30 01:43:21 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:43:21.226794 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2441e9b7-b88e-4116-85a4-2b59182e154e None None] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:21.285184 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7b7a301-5121-4159-b3c9-d3fa23626123 None None] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:21.285556 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7b7a301-5121-4159-b3c9-d3fa23626123 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:21.285746 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7b7a301-5121-4159-b3c9-d3fa23626123 None None] PATH_INFO: `/` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:21.287068 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 273/550] 10.222.0.27 () {54 vars in 927 bytes} [Tue Aug 30 01:43:21 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 30 01:43:21.293766 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f2ce91f-8b66-4e80-8f48-277b9e3f8274 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:21.294047 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f2ce91f-8b66-4e80-8f48-277b9e3f8274 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:21.294316 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f2ce91f-8b66-4e80-8f48-277b9e3f8274 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:21.297809 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:43:21.329090 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-5f2ce91f-8b66-4e80-8f48-277b9e3f8274 None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 01:43:21.330331 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-5f2ce91f-8b66-4e80-8f48-277b9e3f8274 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:43:21.369544 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2441e9b7-b88e-4116-85a4-2b59182e154e None None] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:21.405217 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af634918-5755-4a82-8899-d34186965e1f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:21.405878 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af634918-5755-4a82-8899-d34186965e1f None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:21.406184 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af634918-5755-4a82-8899-d34186965e1f None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:21.406533 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af634918-5755-4a82-8899-d34186965e1f None admin] PATH_INFO: `/v3/users/swift` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:21.414525 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af634918-5755-4a82-8899-d34186965e1f None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:21.416209 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af634918-5755-4a82-8899-d34186965e1f None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:21.428386 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application [None req-af634918-5755-4a82-8899-d34186965e1f None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Aug 30 01:43:21.428386 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 01:43:21.428386 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 01:43:21.428386 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 01:43:21.428386 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 01:43:21.428386 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 01:43:21.428386 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 01:43:21.428386 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 01:43:21.428386 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 01:43:21.428386 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 01:43:21.428386 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 01:43:21.428386 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 01:43:21.428386 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 30 01:43:21.428386 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 30 01:43:21.428386 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 30 01:43:21.428386 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 30 01:43:21.429371 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 01:43:21.429371 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 01:43:21.429371 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 30 01:43:21.429371 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 30 01:43:21.429371 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 30 01:43:21.429371 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 30 01:43:21.429371 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 01:43:21.429371 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 01:43:21.429371 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 01:43:21.429371 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 01:43:21.429371 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 01:43:21.429371 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application with Lock( Aug 30 01:43:21.429371 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 01:43:21.429371 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._enter() Aug 30 01:43:21.429371 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 01:43:21.429371 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 01:43:21.429371 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 01:43:21.429371 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.creator() Aug 30 01:43:21.430311 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 01:43:21.430311 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application created_value = creator( Aug 30 01:43:21.430311 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 30 01:43:21.430311 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 30 01:43:21.430311 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 30 01:43:21.430311 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 30 01:43:21.430311 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 30 01:43:21.430311 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 30 01:43:21.430311 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Aug 30 01:43:21.430311 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application Aug 30 01:43:21.430311 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 278/551] 10.222.0.27 () {56 vars in 1096 bytes} [Tue Aug 30 01:43:21 2022] GET /identity/v3/users/swift => generated 83 bytes in 204 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 01:43:21.437609 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af634918-5755-4a82-8899-d34186965e1f None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:21.525596 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 274/552] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Aug 30 01:43:21 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 232 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Aug 30 01:43:21.527762 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f2ce91f-8b66-4e80-8f48-277b9e3f8274 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:21.553501 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af634918-5755-4a82-8899-d34186965e1f None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:21.566051 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:43:21.607608 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5c4e3bb-e4a4-4cb6-944b-f8e8115fe5e4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:21.608225 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5c4e3bb-e4a4-4cb6-944b-f8e8115fe5e4 None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:21.608504 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5c4e3bb-e4a4-4cb6-944b-f8e8115fe5e4 None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:21.608771 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5c4e3bb-e4a4-4cb6-944b-f8e8115fe5e4 None admin] PATH_INFO: `/v3/users` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:21.609522 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5c4e3bb-e4a4-4cb6-944b-f8e8115fe5e4 None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=81578) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:43:21.609830 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5c4e3bb-e4a4-4cb6-944b-f8e8115fe5e4 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:21.615280 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5c4e3bb-e4a4-4cb6-944b-f8e8115fe5e4 None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:21.654865 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 279/553] 10.222.0.27 () {56 vars in 1135 bytes} [Tue Aug 30 01:43:21 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 217 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 30 01:43:21.660520 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5c4e3bb-e4a4-4cb6-944b-f8e8115fe5e4 None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:21.689010 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f2ce91f-8b66-4e80-8f48-277b9e3f8274 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:21.691501 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:43:21.736146 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcc357db-5bdb-435d-b8e5-1fbedc3df651 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:21.736879 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcc357db-5bdb-435d-b8e5-1fbedc3df651 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:21.737319 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcc357db-5bdb-435d-b8e5-1fbedc3df651 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:21.737701 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcc357db-5bdb-435d-b8e5-1fbedc3df651 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:21.762346 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fcc357db-5bdb-435d-b8e5-1fbedc3df651 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 30 01:43:21.767163 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application [None req-fcc357db-5bdb-435d-b8e5-1fbedc3df651 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Aug 30 01:43:21.767163 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 01:43:21.767163 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 01:43:21.767163 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 01:43:21.767163 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 01:43:21.767163 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 01:43:21.767163 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 01:43:21.767163 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 01:43:21.767163 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 01:43:21.767163 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 01:43:21.767163 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 01:43:21.767163 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 01:43:21.767163 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 30 01:43:21.767163 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 30 01:43:21.767163 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 30 01:43:21.767163 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 30 01:43:21.768085 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 30 01:43:21.768085 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 30 01:43:21.768085 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 30 01:43:21.768085 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 30 01:43:21.768085 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 01:43:21.768085 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 01:43:21.768085 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 01:43:21.768085 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 01:43:21.768085 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 01:43:21.768085 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 01:43:21.768085 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 01:43:21.768085 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application with Lock( Aug 30 01:43:21.768085 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 01:43:21.768085 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._enter() Aug 30 01:43:21.768085 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 01:43:21.768085 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 01:43:21.769580 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 01:43:21.769580 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.creator() Aug 30 01:43:21.769580 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 01:43:21.769580 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application created_value = creator( Aug 30 01:43:21.769580 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 30 01:43:21.769580 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 30 01:43:21.769580 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 30 01:43:21.769580 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 30 01:43:21.769580 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 30 01:43:21.769580 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 30 01:43:21.769580 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Aug 30 01:43:21.769580 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Aug 30 01:43:21.769580 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 275/554] 10.222.0.27 () {56 vars in 1102 bytes} [Tue Aug 30 01:43:21 2022] GET /identity/v3/roles/service => generated 85 bytes in 241 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 01:43:21.774402 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcc357db-5bdb-435d-b8e5-1fbedc3df651 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:21.817103 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5c4e3bb-e4a4-4cb6-944b-f8e8115fe5e4 None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:21.863322 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd90db2d-43f0-496b-9b6b-275a34259c92 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:21.864521 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd90db2d-43f0-496b-9b6b-275a34259c92 None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:21.865012 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd90db2d-43f0-496b-9b6b-275a34259c92 None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:21.866076 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd90db2d-43f0-496b-9b6b-275a34259c92 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:21.920684 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:43:21.949863 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcc357db-5bdb-435d-b8e5-1fbedc3df651 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:21.957114 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd90db2d-43f0-496b-9b6b-275a34259c92 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:21.958711 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd90db2d-43f0-496b-9b6b-275a34259c92 None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:21.985643 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f8a1f05-7a19-49b7-8302-2f8f3cda58a5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:21.986258 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f8a1f05-7a19-49b7-8302-2f8f3cda58a5 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:21.986543 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f8a1f05-7a19-49b7-8302-2f8f3cda58a5 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:21.986817 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f8a1f05-7a19-49b7-8302-2f8f3cda58a5 None admin] PATH_INFO: `/v3/roles` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:21.987483 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f8a1f05-7a19-49b7-8302-2f8f3cda58a5 None admin] RBAC: Adding query filter params (name=service) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:43:21.987785 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f8a1f05-7a19-49b7-8302-2f8f3cda58a5 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:21.989325 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f8a1f05-7a19-49b7-8302-2f8f3cda58a5 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:22.003041 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 276/555] 10.222.0.27 () {56 vars in 1103 bytes} [Tue Aug 30 01:43:21 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 225 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:43:22.003041 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46b4250e-64bd-40b8-a402-3adefbd04cdb None None] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:22.003041 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46b4250e-64bd-40b8-a402-3adefbd04cdb None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:22.003041 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46b4250e-64bd-40b8-a402-3adefbd04cdb None None] PATH_INFO: `/` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:22.003041 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 277/556] 10.222.0.27 () {54 vars in 927 bytes} [Tue Aug 30 01:43:21 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 01:43:22.003041 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46b4250e-64bd-40b8-a402-3adefbd04cdb None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:22.033474 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:43:22.073151 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 280/557] 10.222.0.27 () {58 vars in 1282 bytes} [Tue Aug 30 01:43:21 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 413 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Aug 30 01:43:22.073956 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd90db2d-43f0-496b-9b6b-275a34259c92 None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:22.115574 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46b4250e-64bd-40b8-a402-3adefbd04cdb None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:22.141283 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:43:22.149566 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40d709f0-40ad-4268-9187-3d1b4062a330 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:22.150192 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40d709f0-40ad-4268-9187-3d1b4062a330 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:22.150467 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40d709f0-40ad-4268-9187-3d1b4062a330 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:22.154012 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40d709f0-40ad-4268-9187-3d1b4062a330 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:22.160721 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40d709f0-40ad-4268-9187-3d1b4062a330 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 30 01:43:22.166155 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application [None req-40d709f0-40ad-4268-9187-3d1b4062a330 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Aug 30 01:43:22.166155 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 01:43:22.166155 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 01:43:22.166155 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 01:43:22.166155 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 01:43:22.166155 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 01:43:22.166155 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 01:43:22.166155 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 01:43:22.166155 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 01:43:22.166155 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 01:43:22.166155 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 01:43:22.166155 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 01:43:22.166155 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 30 01:43:22.166155 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 30 01:43:22.166155 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 30 01:43:22.166155 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 30 01:43:22.167040 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 30 01:43:22.167040 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 30 01:43:22.167040 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 30 01:43:22.167040 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 30 01:43:22.167040 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 01:43:22.167040 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 01:43:22.167040 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 01:43:22.167040 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 01:43:22.167040 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 01:43:22.167040 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 01:43:22.167040 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 01:43:22.167040 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application with Lock( Aug 30 01:43:22.167040 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 01:43:22.167040 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._enter() Aug 30 01:43:22.167040 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 01:43:22.167040 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 01:43:22.167869 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 01:43:22.167869 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.creator() Aug 30 01:43:22.167869 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 01:43:22.167869 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application created_value = creator( Aug 30 01:43:22.167869 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 30 01:43:22.167869 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 30 01:43:22.167869 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 30 01:43:22.167869 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 30 01:43:22.167869 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 30 01:43:22.167869 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 30 01:43:22.167869 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Aug 30 01:43:22.167869 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Aug 30 01:43:22.167869 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 278/558] 10.222.0.27 () {56 vars in 1102 bytes} [Tue Aug 30 01:43:22 2022] GET /identity/v3/roles/service => generated 85 bytes in 165 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 01:43:22.168538 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea25cdb6-09f2-4a25-891c-50f8478a4c91 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:22.168977 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea25cdb6-09f2-4a25-891c-50f8478a4c91 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:22.168977 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea25cdb6-09f2-4a25-891c-50f8478a4c91 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:22.177829 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd90db2d-43f0-496b-9b6b-275a34259c92 None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:22.186244 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-ea25cdb6-09f2-4a25-891c-50f8478a4c91 None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 01:43:22.187411 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-ea25cdb6-09f2-4a25-891c-50f8478a4c91 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:43:22.211231 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1379b38d-f644-4756-9bc6-0dfdbe32fdef None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:22.211859 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1379b38d-f644-4756-9bc6-0dfdbe32fdef None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:22.212134 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1379b38d-f644-4756-9bc6-0dfdbe32fdef None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:22.212404 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1379b38d-f644-4756-9bc6-0dfdbe32fdef None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:22.222054 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1379b38d-f644-4756-9bc6-0dfdbe32fdef None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:22.223484 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1379b38d-f644-4756-9bc6-0dfdbe32fdef None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:22.238462 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application [None req-1379b38d-f644-4756-9bc6-0dfdbe32fdef None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 01:43:22.238462 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 01:43:22.238462 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 01:43:22.238462 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 01:43:22.238462 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 01:43:22.238462 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 01:43:22.238462 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 01:43:22.238462 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 01:43:22.238462 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 01:43:22.238462 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 01:43:22.238462 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 01:43:22.238462 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 01:43:22.238462 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 30 01:43:22.238462 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 30 01:43:22.238462 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 30 01:43:22.238462 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 30 01:43:22.239602 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 01:43:22.239602 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 01:43:22.239602 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 01:43:22.239602 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 01:43:22.239602 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 01:43:22.239602 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 01:43:22.239602 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 01:43:22.239602 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application with Lock( Aug 30 01:43:22.239602 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 01:43:22.239602 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._enter() Aug 30 01:43:22.239602 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 01:43:22.239602 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 01:43:22.239602 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 01:43:22.239602 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.creator() Aug 30 01:43:22.239602 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 01:43:22.239602 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application created_value = creator( Aug 30 01:43:22.239602 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 30 01:43:22.240779 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 30 01:43:22.240779 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 01:43:22.240779 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application Aug 30 01:43:22.240779 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 281/559] 10.222.0.27 () {56 vars in 1108 bytes} [Tue Aug 30 01:43:22 2022] GET /identity/v3/domains/Default => generated 87 bytes in 166 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 01:43:22.240779 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1379b38d-f644-4756-9bc6-0dfdbe32fdef None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:22.257586 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 279/560] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Aug 30 01:43:22 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 30 01:43:22.258819 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16f3016d-1017-4bc9-8d5a-95f31c6d2f2f None None] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:22.259103 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16f3016d-1017-4bc9-8d5a-95f31c6d2f2f None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:22.259375 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16f3016d-1017-4bc9-8d5a-95f31c6d2f2f None None] PATH_INFO: `/` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:22.260598 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 280/561] 10.222.0.27 () {54 vars in 927 bytes} [Tue Aug 30 01:43:22 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 01:43:22.262277 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16f3016d-1017-4bc9-8d5a-95f31c6d2f2f None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:22.287647 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:43:22.349842 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1379b38d-f644-4756-9bc6-0dfdbe32fdef None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:22.367948 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:43:22.376965 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16f3016d-1017-4bc9-8d5a-95f31c6d2f2f None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:22.386059 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee66153b-206f-4052-91a5-a00b3e91f7bf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:22.386693 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee66153b-206f-4052-91a5-a00b3e91f7bf None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:22.386974 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee66153b-206f-4052-91a5-a00b3e91f7bf None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:22.387247 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee66153b-206f-4052-91a5-a00b3e91f7bf None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:22.399299 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee66153b-206f-4052-91a5-a00b3e91f7bf None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:22.401053 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee66153b-206f-4052-91a5-a00b3e91f7bf None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:22.415428 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85a2ebb4-01e2-40b5-b1ba-b95bfeb80f35 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:22.416035 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85a2ebb4-01e2-40b5-b1ba-b95bfeb80f35 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:22.416315 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85a2ebb4-01e2-40b5-b1ba-b95bfeb80f35 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:22.416476 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application [None req-ee66153b-206f-4052-91a5-a00b3e91f7bf None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 01:43:22.416476 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 01:43:22.416476 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 01:43:22.416476 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 01:43:22.416476 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 01:43:22.416476 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 01:43:22.416476 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 01:43:22.416476 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 01:43:22.416476 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 01:43:22.416476 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 01:43:22.416476 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 01:43:22.416476 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 01:43:22.416476 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 30 01:43:22.416476 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 30 01:43:22.416476 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 30 01:43:22.416476 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 30 01:43:22.417366 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 01:43:22.417366 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 01:43:22.417366 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 01:43:22.417366 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 01:43:22.417366 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 01:43:22.417366 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 01:43:22.417366 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 01:43:22.417366 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application with Lock( Aug 30 01:43:22.417366 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 01:43:22.417366 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._enter() Aug 30 01:43:22.417366 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 01:43:22.417366 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 01:43:22.417366 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 01:43:22.417366 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.creator() Aug 30 01:43:22.417366 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 01:43:22.417366 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application created_value = creator( Aug 30 01:43:22.417366 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 30 01:43:22.423108 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 30 01:43:22.423108 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 01:43:22.423108 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application Aug 30 01:43:22.423279 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85a2ebb4-01e2-40b5-b1ba-b95bfeb80f35 None admin] PATH_INFO: `/v3/roles` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:22.423279 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85a2ebb4-01e2-40b5-b1ba-b95bfeb80f35 None admin] RBAC: Adding query filter params (name=service) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:43:22.423279 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85a2ebb4-01e2-40b5-b1ba-b95bfeb80f35 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:22.423468 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 282/562] 10.222.0.27 () {56 vars in 1108 bytes} [Tue Aug 30 01:43:22 2022] GET /identity/v3/domains/Default => generated 87 bytes in 178 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 01:43:22.423468 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee66153b-206f-4052-91a5-a00b3e91f7bf None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:22.423591 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85a2ebb4-01e2-40b5-b1ba-b95bfeb80f35 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:22.431768 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 281/563] 10.222.0.27 () {56 vars in 1103 bytes} [Tue Aug 30 01:43:22 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:43:22.433117 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e137162f-102c-46ff-9b2e-a6e963fb4e0b None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:22.433400 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e137162f-102c-46ff-9b2e-a6e963fb4e0b None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:22.433705 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e137162f-102c-46ff-9b2e-a6e963fb4e0b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:22.452749 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-e137162f-102c-46ff-9b2e-a6e963fb4e0b None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 01:43:22.453960 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-e137162f-102c-46ff-9b2e-a6e963fb4e0b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:43:22.530694 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 282/564] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Aug 30 01:43:22 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 30 01:43:22.531544 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e137162f-102c-46ff-9b2e-a6e963fb4e0b None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:22.535209 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee66153b-206f-4052-91a5-a00b3e91f7bf None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:22.542579 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:43:22.570857 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-690d33ae-98ec-40f9-8403-77869e00df24 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:22.571468 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-690d33ae-98ec-40f9-8403-77869e00df24 None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:22.571749 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-690d33ae-98ec-40f9-8403-77869e00df24 None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:22.572018 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-690d33ae-98ec-40f9-8403-77869e00df24 None admin] PATH_INFO: `/v3/domains` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:22.572727 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-690d33ae-98ec-40f9-8403-77869e00df24 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81578) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:43:22.573034 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-690d33ae-98ec-40f9-8403-77869e00df24 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:22.574429 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-690d33ae-98ec-40f9-8403-77869e00df24 None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:22.588237 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 283/565] 10.222.0.27 () {56 vars in 1109 bytes} [Tue Aug 30 01:43:22 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:43:22.589082 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-690d33ae-98ec-40f9-8403-77869e00df24 None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:22.620272 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:43:22.641028 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e137162f-102c-46ff-9b2e-a6e963fb4e0b None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:22.679022 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a40d7200-27ff-4286-84f5-35f49619acb6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:22.679591 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a40d7200-27ff-4286-84f5-35f49619acb6 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:22.680225 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a40d7200-27ff-4286-84f5-35f49619acb6 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:22.680225 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a40d7200-27ff-4286-84f5-35f49619acb6 None admin] PATH_INFO: `/v3/domains` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:22.680804 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a40d7200-27ff-4286-84f5-35f49619acb6 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:43:22.681113 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a40d7200-27ff-4286-84f5-35f49619acb6 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:22.682553 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a40d7200-27ff-4286-84f5-35f49619acb6 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:22.692793 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-690d33ae-98ec-40f9-8403-77869e00df24 None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:22.699708 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 283/566] 10.222.0.27 () {56 vars in 1109 bytes} [Tue Aug 30 01:43:22 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:43:22.699708 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a40d7200-27ff-4286-84f5-35f49619acb6 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:22.736081 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b1583aa-dfb0-4f23-93a3-92ae06db11d3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:22.736709 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b1583aa-dfb0-4f23-93a3-92ae06db11d3 None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:22.736986 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b1583aa-dfb0-4f23-93a3-92ae06db11d3 None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:22.737268 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b1583aa-dfb0-4f23-93a3-92ae06db11d3 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:22.748402 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b1583aa-dfb0-4f23-93a3-92ae06db11d3 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:22.749974 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b1583aa-dfb0-4f23-93a3-92ae06db11d3 None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:22.766496 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application [None req-2b1583aa-dfb0-4f23-93a3-92ae06db11d3 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 01:43:22.766496 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 01:43:22.766496 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 01:43:22.766496 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 01:43:22.766496 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 01:43:22.766496 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 01:43:22.766496 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 01:43:22.766496 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 01:43:22.766496 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 01:43:22.766496 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 01:43:22.766496 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 01:43:22.766496 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 01:43:22.766496 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 30 01:43:22.766496 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 30 01:43:22.766496 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 30 01:43:22.766496 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 30 01:43:22.767460 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 01:43:22.767460 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 01:43:22.767460 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 01:43:22.767460 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 01:43:22.767460 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 01:43:22.767460 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 01:43:22.767460 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 01:43:22.767460 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application with Lock( Aug 30 01:43:22.767460 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 01:43:22.767460 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._enter() Aug 30 01:43:22.767460 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 01:43:22.767460 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 01:43:22.767460 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 01:43:22.767460 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.creator() Aug 30 01:43:22.767460 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 01:43:22.767460 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application created_value = creator( Aug 30 01:43:22.767460 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 30 01:43:22.768382 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 30 01:43:22.768382 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 01:43:22.768382 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application Aug 30 01:43:22.768382 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 284/567] 10.222.0.27 () {56 vars in 1108 bytes} [Tue Aug 30 01:43:22 2022] GET /identity/v3/domains/Default => generated 87 bytes in 179 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 01:43:22.769148 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b1583aa-dfb0-4f23-93a3-92ae06db11d3 None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:22.779668 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:43:22.815788 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a40d7200-27ff-4286-84f5-35f49619acb6 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:22.849589 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b36b7fe-7399-49a1-86d2-a2cc0a4c9d26 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:22.850259 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b36b7fe-7399-49a1-86d2-a2cc0a4c9d26 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:22.850592 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b36b7fe-7399-49a1-86d2-a2cc0a4c9d26 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:22.850870 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b36b7fe-7399-49a1-86d2-a2cc0a4c9d26 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:22.856545 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:43:22.861382 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b36b7fe-7399-49a1-86d2-a2cc0a4c9d26 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 30 01:43:22.867180 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application [None req-1b36b7fe-7399-49a1-86d2-a2cc0a4c9d26 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Aug 30 01:43:22.867180 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 01:43:22.867180 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 01:43:22.867180 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 01:43:22.867180 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 01:43:22.867180 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 01:43:22.867180 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 01:43:22.867180 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 01:43:22.867180 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 01:43:22.867180 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 01:43:22.867180 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 01:43:22.867180 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 01:43:22.867180 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 30 01:43:22.867180 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 30 01:43:22.867180 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 30 01:43:22.867180 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 30 01:43:22.873870 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 30 01:43:22.873870 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 30 01:43:22.873870 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 30 01:43:22.873870 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 30 01:43:22.873870 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 01:43:22.873870 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 01:43:22.873870 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 01:43:22.873870 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 01:43:22.873870 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 01:43:22.873870 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 01:43:22.873870 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 01:43:22.873870 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application with Lock( Aug 30 01:43:22.873870 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 01:43:22.873870 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._enter() Aug 30 01:43:22.873870 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 01:43:22.873870 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 01:43:22.874903 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 01:43:22.874903 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.creator() Aug 30 01:43:22.874903 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 01:43:22.874903 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application created_value = creator( Aug 30 01:43:22.874903 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 30 01:43:22.874903 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 30 01:43:22.874903 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 30 01:43:22.874903 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 30 01:43:22.874903 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 30 01:43:22.874903 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 30 01:43:22.874903 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Aug 30 01:43:22.874903 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Aug 30 01:43:22.874903 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 284/568] 10.222.0.27 () {56 vars in 1102 bytes} [Tue Aug 30 01:43:22 2022] GET /identity/v3/roles/service => generated 85 bytes in 172 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 01:43:22.874903 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b36b7fe-7399-49a1-86d2-a2cc0a4c9d26 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:22.875649 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b1583aa-dfb0-4f23-93a3-92ae06db11d3 None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:22.908274 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4c0a852-cb99-413f-ba8b-673fee7525af None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:22.908891 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4c0a852-cb99-413f-ba8b-673fee7525af None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:22.909164 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4c0a852-cb99-413f-ba8b-673fee7525af None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:22.909430 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4c0a852-cb99-413f-ba8b-673fee7525af None admin] PATH_INFO: `/v3/users/glance` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:22.916899 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4c0a852-cb99-413f-ba8b-673fee7525af None admin] RBAC: Authorizing `identity:get_user(user_id=glance)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:22.918342 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4c0a852-cb99-413f-ba8b-673fee7525af None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:22.930108 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application [None req-d4c0a852-cb99-413f-ba8b-673fee7525af None admin] Could not find user: glance.: keystone.exception.UserNotFound: Could not find user: glance. Aug 30 01:43:22.930108 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 01:43:22.930108 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 01:43:22.930108 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 01:43:22.930108 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 01:43:22.930108 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 01:43:22.930108 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 01:43:22.930108 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 01:43:22.930108 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 01:43:22.930108 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 01:43:22.930108 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 01:43:22.930108 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 01:43:22.930108 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 30 01:43:22.930108 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 30 01:43:22.930108 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 30 01:43:22.930108 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 30 01:43:22.931061 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 01:43:22.931061 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 01:43:22.931061 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 30 01:43:22.931061 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 30 01:43:22.931061 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 30 01:43:22.931061 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 30 01:43:22.931061 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 01:43:22.931061 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 01:43:22.931061 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 01:43:22.931061 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 01:43:22.931061 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 01:43:22.931061 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application with Lock( Aug 30 01:43:22.931061 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 01:43:22.931061 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._enter() Aug 30 01:43:22.931061 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 01:43:22.931061 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 01:43:22.931061 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 01:43:22.931061 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.creator() Aug 30 01:43:22.932011 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 01:43:22.932011 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application created_value = creator( Aug 30 01:43:22.932011 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 30 01:43:22.932011 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 30 01:43:22.932011 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 30 01:43:22.932011 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 30 01:43:22.932011 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 30 01:43:22.932011 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 30 01:43:22.932011 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance. Aug 30 01:43:22.932011 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application Aug 30 01:43:22.932011 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 285/569] 10.222.0.27 () {56 vars in 1099 bytes} [Tue Aug 30 01:43:22 2022] GET /identity/v3/users/glance => generated 84 bytes in 163 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 01:43:22.932011 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4c0a852-cb99-413f-ba8b-673fee7525af None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:22.973587 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b36b7fe-7399-49a1-86d2-a2cc0a4c9d26 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:23.000268 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:43:23.008421 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d9e44e5-0898-4e23-b438-adaae6409713 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:23.009045 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d9e44e5-0898-4e23-b438-adaae6409713 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:23.011341 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d9e44e5-0898-4e23-b438-adaae6409713 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:23.011341 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d9e44e5-0898-4e23-b438-adaae6409713 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:23.016212 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d9e44e5-0898-4e23-b438-adaae6409713 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:23.017724 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d9e44e5-0898-4e23-b438-adaae6409713 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:23.029538 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4c0a852-cb99-413f-ba8b-673fee7525af None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:23.029680 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application [None req-7d9e44e5-0898-4e23-b438-adaae6409713 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 30 01:43:23.029680 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 01:43:23.029680 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 01:43:23.029680 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 01:43:23.029680 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 01:43:23.029680 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 01:43:23.029680 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 01:43:23.029680 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 01:43:23.029680 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 01:43:23.029680 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 01:43:23.029680 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 01:43:23.029680 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 01:43:23.029680 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 30 01:43:23.029680 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 30 01:43:23.029680 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 30 01:43:23.030677 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 30 01:43:23.030677 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 01:43:23.030677 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 01:43:23.030677 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 01:43:23.030677 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 01:43:23.030677 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 01:43:23.030677 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 01:43:23.030677 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 01:43:23.030677 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application with Lock( Aug 30 01:43:23.030677 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 01:43:23.030677 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._enter() Aug 30 01:43:23.030677 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 01:43:23.030677 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 01:43:23.030677 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 01:43:23.030677 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.creator() Aug 30 01:43:23.030677 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 01:43:23.030677 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application created_value = creator( Aug 30 01:43:23.031604 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 30 01:43:23.031604 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 30 01:43:23.031604 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 30 01:43:23.031604 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 30 01:43:23.031604 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 30 01:43:23.031604 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 30 01:43:23.031604 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 30 01:43:23.031604 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Aug 30 01:43:23.035477 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 285/570] 10.222.0.27 () {56 vars in 1111 bytes} [Tue Aug 30 01:43:22 2022] GET /identity/v3/projects/service => generated 88 bytes in 167 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 01:43:23.036251 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d9e44e5-0898-4e23-b438-adaae6409713 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:23.062741 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-035caafc-7e66-41aa-a47d-718ef89bebf7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:23.063342 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-035caafc-7e66-41aa-a47d-718ef89bebf7 None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:23.063622 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-035caafc-7e66-41aa-a47d-718ef89bebf7 None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:23.063892 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-035caafc-7e66-41aa-a47d-718ef89bebf7 None admin] PATH_INFO: `/v3/domains` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:23.064527 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-035caafc-7e66-41aa-a47d-718ef89bebf7 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81578) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:43:23.064832 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-035caafc-7e66-41aa-a47d-718ef89bebf7 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:23.066313 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-035caafc-7e66-41aa-a47d-718ef89bebf7 None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:23.078242 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 286/571] 10.222.0.27 () {56 vars in 1109 bytes} [Tue Aug 30 01:43:22 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:43:23.079027 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-035caafc-7e66-41aa-a47d-718ef89bebf7 None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:23.081500 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:43:23.142462 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d9e44e5-0898-4e23-b438-adaae6409713 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:23.175648 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38772dd3-d032-411e-80d9-124f72f7b0f4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:23.176272 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38772dd3-d032-411e-80d9-124f72f7b0f4 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:23.176545 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38772dd3-d032-411e-80d9-124f72f7b0f4 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:23.176816 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38772dd3-d032-411e-80d9-124f72f7b0f4 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:23.179426 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-035caafc-7e66-41aa-a47d-718ef89bebf7 None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:23.187137 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38772dd3-d032-411e-80d9-124f72f7b0f4 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 30 01:43:23.192402 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application [None req-38772dd3-d032-411e-80d9-124f72f7b0f4 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Aug 30 01:43:23.192402 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 01:43:23.192402 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 01:43:23.192402 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 01:43:23.192402 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 01:43:23.192402 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 01:43:23.192402 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 01:43:23.192402 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 01:43:23.192402 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 01:43:23.192402 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 01:43:23.192402 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 01:43:23.192402 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 01:43:23.192402 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 30 01:43:23.192402 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 30 01:43:23.192402 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 30 01:43:23.192402 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 30 01:43:23.195129 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 30 01:43:23.195129 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 30 01:43:23.195129 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 30 01:43:23.195129 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 30 01:43:23.195129 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 01:43:23.195129 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 01:43:23.195129 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 01:43:23.195129 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 01:43:23.195129 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 01:43:23.195129 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 01:43:23.195129 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 01:43:23.195129 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application with Lock( Aug 30 01:43:23.195129 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 01:43:23.195129 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._enter() Aug 30 01:43:23.195129 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 01:43:23.195129 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 01:43:23.196128 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 01:43:23.196128 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.creator() Aug 30 01:43:23.196128 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 01:43:23.196128 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application created_value = creator( Aug 30 01:43:23.196128 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 30 01:43:23.196128 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 30 01:43:23.196128 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 30 01:43:23.196128 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 30 01:43:23.196128 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 30 01:43:23.196128 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 30 01:43:23.196128 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Aug 30 01:43:23.196128 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Aug 30 01:43:23.196128 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 286/572] 10.222.0.27 () {56 vars in 1102 bytes} [Tue Aug 30 01:43:23 2022] GET /identity/v3/roles/service => generated 85 bytes in 158 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 01:43:23.196128 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38772dd3-d032-411e-80d9-124f72f7b0f4 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:23.217876 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-224cc441-16a2-426f-98a2-497fd2fd413a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:23.218470 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-224cc441-16a2-426f-98a2-497fd2fd413a None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:23.218739 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-224cc441-16a2-426f-98a2-497fd2fd413a None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:23.219004 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-224cc441-16a2-426f-98a2-497fd2fd413a None admin] PATH_INFO: `/v3/roles` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:23.219657 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-224cc441-16a2-426f-98a2-497fd2fd413a None admin] RBAC: Adding query filter params (name=service) {{(pid=81578) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:43:23.219952 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-224cc441-16a2-426f-98a2-497fd2fd413a None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:23.221291 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-224cc441-16a2-426f-98a2-497fd2fd413a None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:23.229229 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 287/573] 10.222.0.27 () {56 vars in 1103 bytes} [Tue Aug 30 01:43:23 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:43:23.230019 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-224cc441-16a2-426f-98a2-497fd2fd413a None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:23.245904 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:43:23.293197 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:43:23.296377 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38772dd3-d032-411e-80d9-124f72f7b0f4 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:23.329642 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f372ae20-5530-432c-8c81-84109d6861a9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:23.330084 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-224cc441-16a2-426f-98a2-497fd2fd413a None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:23.330238 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f372ae20-5530-432c-8c81-84109d6861a9 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:23.330513 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f372ae20-5530-432c-8c81-84109d6861a9 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:23.330781 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f372ae20-5530-432c-8c81-84109d6861a9 None admin] PATH_INFO: `/v3/users` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:23.331495 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f372ae20-5530-432c-8c81-84109d6861a9 None admin] RBAC: Adding query filter params (domain_id=default, name=glance) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:43:23.331795 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f372ae20-5530-432c-8c81-84109d6861a9 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:23.333253 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f372ae20-5530-432c-8c81-84109d6861a9 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:23.350656 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 287/574] 10.222.0.27 () {56 vars in 1137 bytes} [Tue Aug 30 01:43:23 2022] GET /identity/v3/users?domain_id=default&name=glance => generated 372 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:43:23.356022 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f372ae20-5530-432c-8c81-84109d6861a9 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:23.363505 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb7c61d9-f295-4caa-b02f-936a6d92c6ea None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:23.364110 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb7c61d9-f295-4caa-b02f-936a6d92c6ea None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:23.364385 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb7c61d9-f295-4caa-b02f-936a6d92c6ea None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:23.364652 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb7c61d9-f295-4caa-b02f-936a6d92c6ea None admin] PATH_INFO: `/v3/projects` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:23.365335 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb7c61d9-f295-4caa-b02f-936a6d92c6ea None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81578) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:43:23.365671 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb7c61d9-f295-4caa-b02f-936a6d92c6ea None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:23.367176 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb7c61d9-f295-4caa-b02f-936a6d92c6ea None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:23.378741 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 288/575] 10.222.0.27 () {56 vars in 1148 bytes} [Tue Aug 30 01:43:23 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:43:23.379582 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb7c61d9-f295-4caa-b02f-936a6d92c6ea None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:23.454293 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f372ae20-5530-432c-8c81-84109d6861a9 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:23.461445 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:43:23.480499 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb7c61d9-f295-4caa-b02f-936a6d92c6ea None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:23.487410 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9aca325c-9b7a-4512-9e49-a8beeeaa9428 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:23.488105 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9aca325c-9b7a-4512-9e49-a8beeeaa9428 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:23.488380 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9aca325c-9b7a-4512-9e49-a8beeeaa9428 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:23.488648 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9aca325c-9b7a-4512-9e49-a8beeeaa9428 None admin] PATH_INFO: `/v3/users/nova` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:23.497101 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9aca325c-9b7a-4512-9e49-a8beeeaa9428 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:23.498585 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9aca325c-9b7a-4512-9e49-a8beeeaa9428 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:23.503906 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:43:23.510402 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application [None req-9aca325c-9b7a-4512-9e49-a8beeeaa9428 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Aug 30 01:43:23.510402 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 01:43:23.510402 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 01:43:23.510402 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 01:43:23.510402 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 01:43:23.510402 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 01:43:23.510402 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 01:43:23.510402 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 01:43:23.510402 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 01:43:23.510402 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 01:43:23.510402 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 01:43:23.510402 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 01:43:23.510402 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 30 01:43:23.510402 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 30 01:43:23.510402 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 30 01:43:23.510402 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 30 01:43:23.511501 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 01:43:23.511501 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 01:43:23.511501 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 30 01:43:23.511501 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 30 01:43:23.511501 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 30 01:43:23.511501 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 30 01:43:23.511501 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 01:43:23.511501 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 01:43:23.511501 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 01:43:23.511501 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 01:43:23.511501 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 01:43:23.511501 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application with Lock( Aug 30 01:43:23.511501 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 01:43:23.511501 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._enter() Aug 30 01:43:23.511501 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 01:43:23.511501 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 01:43:23.511501 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 01:43:23.511501 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.creator() Aug 30 01:43:23.512442 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 01:43:23.512442 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application created_value = creator( Aug 30 01:43:23.512442 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 30 01:43:23.512442 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 30 01:43:23.512442 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 30 01:43:23.512442 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 30 01:43:23.512442 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 30 01:43:23.512442 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 30 01:43:23.512442 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Aug 30 01:43:23.512442 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Aug 30 01:43:23.512442 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 288/576] 10.222.0.27 () {56 vars in 1093 bytes} [Tue Aug 30 01:43:23 2022] GET /identity/v3/users/nova => generated 82 bytes in 160 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 01:43:23.512442 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9aca325c-9b7a-4512-9e49-a8beeeaa9428 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:23.516921 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7408834d-91e6-4a8e-8500-8453eb6501b6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:23.517556 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7408834d-91e6-4a8e-8500-8453eb6501b6 None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:23.517842 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7408834d-91e6-4a8e-8500-8453eb6501b6 None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:23.518114 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7408834d-91e6-4a8e-8500-8453eb6501b6 None admin] PATH_INFO: `/v3/roles` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:23.518781 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7408834d-91e6-4a8e-8500-8453eb6501b6 None admin] RBAC: Adding query filter params (name=service) {{(pid=81578) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:43:23.519082 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7408834d-91e6-4a8e-8500-8453eb6501b6 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:23.520436 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7408834d-91e6-4a8e-8500-8453eb6501b6 None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:23.528476 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 289/577] 10.222.0.27 () {56 vars in 1103 bytes} [Tue Aug 30 01:43:23 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:43:23.529261 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7408834d-91e6-4a8e-8500-8453eb6501b6 None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:23.612238 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9aca325c-9b7a-4512-9e49-a8beeeaa9428 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:23.629089 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7408834d-91e6-4a8e-8500-8453eb6501b6 None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:23.646831 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd0c4a05-fbf7-43e6-83c8-dd61a8a355a2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:23.647636 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd0c4a05-fbf7-43e6-83c8-dd61a8a355a2 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:23.647915 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd0c4a05-fbf7-43e6-83c8-dd61a8a355a2 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:23.648181 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd0c4a05-fbf7-43e6-83c8-dd61a8a355a2 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:23.658610 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd0c4a05-fbf7-43e6-83c8-dd61a8a355a2 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:23.660060 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd0c4a05-fbf7-43e6-83c8-dd61a8a355a2 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:23.662129 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-582ca48c-3b52-4b1b-add2-2f193fd00859 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:23.662893 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-582ca48c-3b52-4b1b-add2-2f193fd00859 None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:23.663237 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-582ca48c-3b52-4b1b-add2-2f193fd00859 None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:23.663501 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-582ca48c-3b52-4b1b-add2-2f193fd00859 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:23.673853 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-582ca48c-3b52-4b1b-add2-2f193fd00859 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:23.673996 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application [None req-fd0c4a05-fbf7-43e6-83c8-dd61a8a355a2 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 01:43:23.673996 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 01:43:23.673996 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 01:43:23.673996 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 01:43:23.673996 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 01:43:23.673996 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 01:43:23.673996 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 01:43:23.673996 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 01:43:23.673996 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 01:43:23.673996 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 01:43:23.673996 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 01:43:23.673996 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 01:43:23.673996 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 30 01:43:23.673996 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 30 01:43:23.673996 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 30 01:43:23.673996 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 30 01:43:23.675068 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 01:43:23.675068 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 01:43:23.675068 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 01:43:23.675068 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 01:43:23.675068 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 01:43:23.675068 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 01:43:23.675068 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 01:43:23.675068 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application with Lock( Aug 30 01:43:23.675068 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 01:43:23.675068 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._enter() Aug 30 01:43:23.675068 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 01:43:23.675068 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 01:43:23.675068 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 01:43:23.675068 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.creator() Aug 30 01:43:23.675068 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 01:43:23.675068 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application created_value = creator( Aug 30 01:43:23.675068 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 30 01:43:23.676139 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 30 01:43:23.676139 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 01:43:23.676139 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Aug 30 01:43:23.676139 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 289/578] 10.222.0.27 () {56 vars in 1108 bytes} [Tue Aug 30 01:43:23 2022] GET /identity/v3/domains/Default => generated 87 bytes in 163 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 01:43:23.676407 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd0c4a05-fbf7-43e6-83c8-dd61a8a355a2 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:23.676947 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-582ca48c-3b52-4b1b-add2-2f193fd00859 None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:23.693153 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application [None req-582ca48c-3b52-4b1b-add2-2f193fd00859 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 01:43:23.693153 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 01:43:23.693153 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 01:43:23.693153 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 01:43:23.693153 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 01:43:23.693153 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 01:43:23.693153 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 01:43:23.693153 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 01:43:23.693153 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 01:43:23.693153 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 01:43:23.693153 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 01:43:23.693153 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 01:43:23.693153 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 30 01:43:23.693153 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 30 01:43:23.693153 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 30 01:43:23.693153 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 30 01:43:23.697162 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 01:43:23.697162 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 01:43:23.697162 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 01:43:23.697162 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 01:43:23.697162 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 01:43:23.697162 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 01:43:23.697162 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 01:43:23.697162 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application with Lock( Aug 30 01:43:23.697162 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 01:43:23.697162 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._enter() Aug 30 01:43:23.697162 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 01:43:23.697162 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 01:43:23.697162 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 01:43:23.697162 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.creator() Aug 30 01:43:23.697162 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 01:43:23.697162 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application created_value = creator( Aug 30 01:43:23.697162 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 30 01:43:23.698298 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 30 01:43:23.698298 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 01:43:23.698298 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application Aug 30 01:43:23.698298 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 290/579] 10.222.0.27 () {56 vars in 1108 bytes} [Tue Aug 30 01:43:23 2022] GET /identity/v3/domains/Default => generated 87 bytes in 166 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 01:43:23.698298 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-582ca48c-3b52-4b1b-add2-2f193fd00859 None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:23.701922 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:43:23.737503 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:43:23.784779 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd0c4a05-fbf7-43e6-83c8-dd61a8a355a2 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:23.797655 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-582ca48c-3b52-4b1b-add2-2f193fd00859 None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:23.817638 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3dd6934c-84f5-473a-bba8-f64ff098d97a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:23.818317 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dd6934c-84f5-473a-bba8-f64ff098d97a None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:23.818671 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dd6934c-84f5-473a-bba8-f64ff098d97a None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:23.819552 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dd6934c-84f5-473a-bba8-f64ff098d97a None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:23.820276 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3dd6934c-84f5-473a-bba8-f64ff098d97a None admin] RBAC: Adding query filter params (role.id=aa59b0c9009a480994d9df81a5f8d404, scope.project.id=4562f3951fbe465280073e60a11e44e2, user.id=b4f0008fde424f22bc1f7307cf422774) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:43:23.820586 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3dd6934c-84f5-473a-bba8-f64ff098d97a None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:23.822037 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3dd6934c-84f5-473a-bba8-f64ff098d97a None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:23.827770 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 290/580] 10.222.0.27 () {56 vars in 1374 bytes} [Tue Aug 30 01:43:23 2022] GET /identity/v3/role_assignments?user.id=b4f0008fde424f22bc1f7307cf422774&scope.project.id=4562f3951fbe465280073e60a11e44e2&role.id=aa59b0c9009a480994d9df81a5f8d404 => generated 259 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:43:23.828559 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3dd6934c-84f5-473a-bba8-f64ff098d97a None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:23.833931 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-879b85d0-da0e-4000-97e2-f47a670894ad None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:23.835724 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-879b85d0-da0e-4000-97e2-f47a670894ad None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:23.836152 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-879b85d0-da0e-4000-97e2-f47a670894ad None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:23.836575 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-879b85d0-da0e-4000-97e2-f47a670894ad None admin] PATH_INFO: `/v3/users` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:23.837334 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-879b85d0-da0e-4000-97e2-f47a670894ad None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81578) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:43:23.837674 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-879b85d0-da0e-4000-97e2-f47a670894ad None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:23.839136 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-879b85d0-da0e-4000-97e2-f47a670894ad None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:23.860165 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 291/581] 10.222.0.27 () {56 vars in 1133 bytes} [Tue Aug 30 01:43:23 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:43:23.862707 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-879b85d0-da0e-4000-97e2-f47a670894ad None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:23.926570 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:43:23.947027 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3dd6934c-84f5-473a-bba8-f64ff098d97a None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:23.972675 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:43:23.980898 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-879b85d0-da0e-4000-97e2-f47a670894ad None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:23.994442 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4a515cf-2501-4bf6-9601-c6c158cf0e03 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:23.994442 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4a515cf-2501-4bf6-9601-c6c158cf0e03 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:23.994442 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4a515cf-2501-4bf6-9601-c6c158cf0e03 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:23.994442 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4a515cf-2501-4bf6-9601-c6c158cf0e03 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:23.999984 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4a515cf-2501-4bf6-9601-c6c158cf0e03 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:24.001504 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4a515cf-2501-4bf6-9601-c6c158cf0e03 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:24.017083 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-655b769e-065b-4209-9bdd-22398e3a5054 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:24.017577 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application [None req-b4a515cf-2501-4bf6-9601-c6c158cf0e03 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 01:43:24.017577 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 01:43:24.017577 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 01:43:24.017577 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 01:43:24.017577 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 01:43:24.017577 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 01:43:24.017577 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 01:43:24.017577 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 01:43:24.017577 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 01:43:24.017577 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 01:43:24.017577 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 01:43:24.017577 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 01:43:24.017577 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 30 01:43:24.017577 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 30 01:43:24.017577 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 30 01:43:24.017577 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 30 01:43:24.018508 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 01:43:24.018508 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 01:43:24.018508 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 01:43:24.018508 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 01:43:24.018508 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 01:43:24.018508 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 01:43:24.018508 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 01:43:24.018508 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application with Lock( Aug 30 01:43:24.018508 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 01:43:24.018508 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._enter() Aug 30 01:43:24.018508 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 01:43:24.018508 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 01:43:24.018508 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 01:43:24.018508 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.creator() Aug 30 01:43:24.018508 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 01:43:24.018508 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application created_value = creator( Aug 30 01:43:24.018508 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 30 01:43:24.019429 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 30 01:43:24.019429 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 01:43:24.019429 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Aug 30 01:43:24.019598 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-655b769e-065b-4209-9bdd-22398e3a5054 None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:24.019598 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-655b769e-065b-4209-9bdd-22398e3a5054 None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:24.019598 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-655b769e-065b-4209-9bdd-22398e3a5054 None admin] PATH_INFO: `/v3/domains` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:24.019794 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 291/582] 10.222.0.27 () {56 vars in 1108 bytes} [Tue Aug 30 01:43:23 2022] GET /identity/v3/domains/Default => generated 87 bytes in 191 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 01:43:24.020388 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4a515cf-2501-4bf6-9601-c6c158cf0e03 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:24.024092 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-655b769e-065b-4209-9bdd-22398e3a5054 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81578) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:43:24.024437 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-655b769e-065b-4209-9bdd-22398e3a5054 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:24.025867 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-655b769e-065b-4209-9bdd-22398e3a5054 None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:24.038837 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 292/583] 10.222.0.27 () {56 vars in 1109 bytes} [Tue Aug 30 01:43:23 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:43:24.040089 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-655b769e-065b-4209-9bdd-22398e3a5054 None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:24.146156 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4a515cf-2501-4bf6-9601-c6c158cf0e03 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:24.146510 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-655b769e-065b-4209-9bdd-22398e3a5054 None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:24.185237 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b808f841-7e89-454f-80fe-7f1d138c3f20 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:24.185499 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2839b45a-207e-4db1-b402-7790328c38f0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:24.185894 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b808f841-7e89-454f-80fe-7f1d138c3f20 None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:24.186019 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2839b45a-207e-4db1-b402-7790328c38f0 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:24.186185 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b808f841-7e89-454f-80fe-7f1d138c3f20 None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:24.186281 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2839b45a-207e-4db1-b402-7790328c38f0 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:24.186462 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b808f841-7e89-454f-80fe-7f1d138c3f20 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:24.186573 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2839b45a-207e-4db1-b402-7790328c38f0 None admin] PATH_INFO: `/v3/domains` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:24.187121 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2839b45a-207e-4db1-b402-7790328c38f0 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:43:24.187391 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2839b45a-207e-4db1-b402-7790328c38f0 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:24.188757 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2839b45a-207e-4db1-b402-7790328c38f0 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:24.190378 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:43:24.197886 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b808f841-7e89-454f-80fe-7f1d138c3f20 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:24.199326 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b808f841-7e89-454f-80fe-7f1d138c3f20 None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:24.205637 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 292/584] 10.222.0.27 () {56 vars in 1109 bytes} [Tue Aug 30 01:43:24 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:43:24.205637 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2839b45a-207e-4db1-b402-7790328c38f0 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:24.221208 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application [None req-b808f841-7e89-454f-80fe-7f1d138c3f20 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 01:43:24.221208 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 01:43:24.221208 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 01:43:24.221208 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 01:43:24.221208 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 01:43:24.221208 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 01:43:24.221208 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 01:43:24.221208 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 01:43:24.221208 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 01:43:24.221208 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 01:43:24.221208 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 01:43:24.221208 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 01:43:24.221208 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 30 01:43:24.221208 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 30 01:43:24.221208 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 30 01:43:24.221208 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 30 01:43:24.222465 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 01:43:24.222465 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 01:43:24.222465 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 01:43:24.222465 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 01:43:24.222465 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 01:43:24.222465 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 01:43:24.222465 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 01:43:24.222465 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application with Lock( Aug 30 01:43:24.222465 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 01:43:24.222465 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._enter() Aug 30 01:43:24.222465 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 01:43:24.222465 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 01:43:24.222465 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 01:43:24.222465 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.creator() Aug 30 01:43:24.222465 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 01:43:24.222465 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application created_value = creator( Aug 30 01:43:24.222465 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 30 01:43:24.223581 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 30 01:43:24.223581 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 01:43:24.223581 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application Aug 30 01:43:24.228266 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 293/585] 10.222.0.27 () {56 vars in 1108 bytes} [Tue Aug 30 01:43:24 2022] GET /identity/v3/domains/Default => generated 87 bytes in 188 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 01:43:24.228935 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b808f841-7e89-454f-80fe-7f1d138c3f20 None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:24.240509 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:43:24.315852 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2839b45a-207e-4db1-b402-7790328c38f0 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:24.331999 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b808f841-7e89-454f-80fe-7f1d138c3f20 None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:24.361589 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b86d75e-2d1e-4d60-bd60-b343554d02ed None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:24.362547 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b86d75e-2d1e-4d60-bd60-b343554d02ed None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:24.363004 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b86d75e-2d1e-4d60-bd60-b343554d02ed None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:24.363452 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b86d75e-2d1e-4d60-bd60-b343554d02ed None admin] PATH_INFO: `/v3/domains` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:24.364387 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b86d75e-2d1e-4d60-bd60-b343554d02ed None admin] RBAC: Adding query filter params (name=Default) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:43:24.364785 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b86d75e-2d1e-4d60-bd60-b343554d02ed None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:24.367670 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b86d75e-2d1e-4d60-bd60-b343554d02ed None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:24.385807 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-409e4f39-b5a7-4a51-b401-38535527c241 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:24.390810 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 293/586] 10.222.0.27 () {56 vars in 1109 bytes} [Tue Aug 30 01:43:24 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:43:24.390932 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-409e4f39-b5a7-4a51-b401-38535527c241 None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:24.390932 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-409e4f39-b5a7-4a51-b401-38535527c241 None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:24.391095 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b86d75e-2d1e-4d60-bd60-b343554d02ed None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:24.391177 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-409e4f39-b5a7-4a51-b401-38535527c241 None admin] PATH_INFO: `/v3/users/cinder` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:24.400533 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-409e4f39-b5a7-4a51-b401-38535527c241 None admin] RBAC: Authorizing `identity:get_user(user_id=cinder)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:24.402171 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-409e4f39-b5a7-4a51-b401-38535527c241 None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:24.413953 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application [None req-409e4f39-b5a7-4a51-b401-38535527c241 None admin] Could not find user: cinder.: keystone.exception.UserNotFound: Could not find user: cinder. Aug 30 01:43:24.413953 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 01:43:24.413953 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 01:43:24.413953 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 01:43:24.413953 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 01:43:24.413953 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 01:43:24.413953 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 01:43:24.413953 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 01:43:24.413953 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 01:43:24.413953 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 01:43:24.413953 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 01:43:24.413953 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 01:43:24.413953 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 30 01:43:24.413953 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 30 01:43:24.413953 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 30 01:43:24.413953 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 30 01:43:24.415087 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 01:43:24.415087 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 01:43:24.415087 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 30 01:43:24.415087 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 30 01:43:24.415087 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 30 01:43:24.415087 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 30 01:43:24.415087 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 01:43:24.415087 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 01:43:24.415087 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 01:43:24.415087 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 01:43:24.415087 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 01:43:24.415087 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application with Lock( Aug 30 01:43:24.415087 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 01:43:24.415087 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._enter() Aug 30 01:43:24.415087 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 01:43:24.415087 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 01:43:24.415087 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 01:43:24.415087 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.creator() Aug 30 01:43:24.416159 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 01:43:24.416159 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application created_value = creator( Aug 30 01:43:24.416159 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 30 01:43:24.416159 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 30 01:43:24.416159 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 30 01:43:24.416159 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 30 01:43:24.416159 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 30 01:43:24.416159 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 30 01:43:24.416159 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: cinder. Aug 30 01:43:24.416159 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application Aug 30 01:43:24.416159 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 294/587] 10.222.0.27 () {56 vars in 1099 bytes} [Tue Aug 30 01:43:24 2022] GET /identity/v3/users/cinder => generated 84 bytes in 187 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 01:43:24.420981 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-409e4f39-b5a7-4a51-b401-38535527c241 None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:24.450420 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:43:24.496135 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:43:24.497482 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b86d75e-2d1e-4d60-bd60-b343554d02ed None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:24.529411 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-409e4f39-b5a7-4a51-b401-38535527c241 None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:24.532714 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-952215c2-9485-4807-9586-c941fd7c8bcb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:24.533341 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-952215c2-9485-4807-9586-c941fd7c8bcb None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:24.533647 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-952215c2-9485-4807-9586-c941fd7c8bcb None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:24.533917 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-952215c2-9485-4807-9586-c941fd7c8bcb None admin] PATH_INFO: `/v3/projects/service` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:24.541212 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-952215c2-9485-4807-9586-c941fd7c8bcb None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:24.542690 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-952215c2-9485-4807-9586-c941fd7c8bcb None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:24.553869 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application [None req-952215c2-9485-4807-9586-c941fd7c8bcb None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 30 01:43:24.553869 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 01:43:24.553869 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 01:43:24.553869 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 01:43:24.553869 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 01:43:24.553869 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 01:43:24.553869 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 01:43:24.553869 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 01:43:24.553869 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 01:43:24.553869 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 01:43:24.553869 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 01:43:24.553869 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 01:43:24.553869 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 30 01:43:24.553869 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 30 01:43:24.553869 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 30 01:43:24.554760 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 30 01:43:24.554760 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 01:43:24.554760 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 01:43:24.554760 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 01:43:24.554760 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 01:43:24.554760 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 01:43:24.554760 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 01:43:24.554760 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 01:43:24.554760 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application with Lock( Aug 30 01:43:24.554760 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 01:43:24.554760 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._enter() Aug 30 01:43:24.554760 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 01:43:24.554760 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 01:43:24.554760 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 01:43:24.554760 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.creator() Aug 30 01:43:24.554760 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 01:43:24.554760 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application created_value = creator( Aug 30 01:43:24.555654 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 30 01:43:24.555654 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 30 01:43:24.555654 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 30 01:43:24.555654 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 30 01:43:24.555654 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 30 01:43:24.555654 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 30 01:43:24.555654 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 30 01:43:24.555654 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Aug 30 01:43:24.555654 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 294/588] 10.222.0.27 () {56 vars in 1111 bytes} [Tue Aug 30 01:43:24 2022] GET /identity/v3/projects/service => generated 88 bytes in 168 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 01:43:24.556241 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-952215c2-9485-4807-9586-c941fd7c8bcb None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:24.571048 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc1e90b0-32d3-4fd1-a2fd-098dc810e708 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:24.571798 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc1e90b0-32d3-4fd1-a2fd-098dc810e708 None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:24.572089 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc1e90b0-32d3-4fd1-a2fd-098dc810e708 None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:24.572362 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc1e90b0-32d3-4fd1-a2fd-098dc810e708 None admin] PATH_INFO: `/v3/domains` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:24.573389 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc1e90b0-32d3-4fd1-a2fd-098dc810e708 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81578) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:43:24.573870 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc1e90b0-32d3-4fd1-a2fd-098dc810e708 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:24.575225 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc1e90b0-32d3-4fd1-a2fd-098dc810e708 None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:24.592039 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 295/589] 10.222.0.27 () {56 vars in 1109 bytes} [Tue Aug 30 01:43:24 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:43:24.592039 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc1e90b0-32d3-4fd1-a2fd-098dc810e708 None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:24.663304 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-952215c2-9485-4807-9586-c941fd7c8bcb None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:24.677736 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:43:24.696174 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc1e90b0-32d3-4fd1-a2fd-098dc810e708 None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:24.698470 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8f28eaa-034c-401e-8568-efef04e74762 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:24.699184 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8f28eaa-034c-401e-8568-efef04e74762 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:24.699466 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8f28eaa-034c-401e-8568-efef04e74762 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:24.699741 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8f28eaa-034c-401e-8568-efef04e74762 None admin] PATH_INFO: `/v3/users/neutron` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:24.707881 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8f28eaa-034c-401e-8568-efef04e74762 None admin] RBAC: Authorizing `identity:get_user(user_id=neutron)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:24.709325 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8f28eaa-034c-401e-8568-efef04e74762 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:24.724210 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application [None req-c8f28eaa-034c-401e-8568-efef04e74762 None admin] Could not find user: neutron.: keystone.exception.UserNotFound: Could not find user: neutron. Aug 30 01:43:24.724210 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 01:43:24.724210 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 01:43:24.724210 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 01:43:24.724210 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 01:43:24.724210 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 01:43:24.724210 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 01:43:24.724210 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 01:43:24.724210 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 01:43:24.724210 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 01:43:24.724210 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 01:43:24.724210 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 01:43:24.724210 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 30 01:43:24.724210 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 30 01:43:24.724210 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 30 01:43:24.724210 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 30 01:43:24.725814 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 01:43:24.725814 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 01:43:24.725814 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 30 01:43:24.725814 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 30 01:43:24.725814 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 30 01:43:24.725814 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 30 01:43:24.725814 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 01:43:24.725814 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 01:43:24.725814 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 01:43:24.725814 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 01:43:24.725814 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 01:43:24.725814 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application with Lock( Aug 30 01:43:24.725814 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 01:43:24.725814 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._enter() Aug 30 01:43:24.725814 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 01:43:24.725814 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 01:43:24.725814 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 01:43:24.725814 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.creator() Aug 30 01:43:24.727297 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 01:43:24.727297 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application created_value = creator( Aug 30 01:43:24.727297 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 30 01:43:24.727297 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 30 01:43:24.727297 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 30 01:43:24.727297 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 30 01:43:24.727297 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 30 01:43:24.727297 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 30 01:43:24.727297 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: neutron. Aug 30 01:43:24.727297 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Aug 30 01:43:24.727297 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 295/590] 10.222.0.27 () {56 vars in 1102 bytes} [Tue Aug 30 01:43:24 2022] GET /identity/v3/users/neutron => generated 85 bytes in 170 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 01:43:24.727297 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8f28eaa-034c-401e-8568-efef04e74762 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:24.738764 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:43:24.752041 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fc5b68d-011c-4145-9ed1-fc862b9cd187 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:24.752041 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fc5b68d-011c-4145-9ed1-fc862b9cd187 None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:24.752041 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fc5b68d-011c-4145-9ed1-fc862b9cd187 None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:24.752041 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fc5b68d-011c-4145-9ed1-fc862b9cd187 None admin] PATH_INFO: `/v3/users` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:24.752428 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fc5b68d-011c-4145-9ed1-fc862b9cd187 None admin] RBAC: Adding query filter params (domain_id=default, name=cinder) {{(pid=81578) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:43:24.752428 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fc5b68d-011c-4145-9ed1-fc862b9cd187 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:24.752428 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fc5b68d-011c-4145-9ed1-fc862b9cd187 None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:24.771203 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 296/591] 10.222.0.27 () {56 vars in 1137 bytes} [Tue Aug 30 01:43:24 2022] GET /identity/v3/users?domain_id=default&name=cinder => generated 372 bytes in 183 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:43:24.776681 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fc5b68d-011c-4145-9ed1-fc862b9cd187 None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:24.847290 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8f28eaa-034c-401e-8568-efef04e74762 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:24.884401 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc4fda7d-99a5-472a-908a-b0fa2b82656f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:24.885169 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc4fda7d-99a5-472a-908a-b0fa2b82656f None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:24.885617 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc4fda7d-99a5-472a-908a-b0fa2b82656f None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:24.885905 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc4fda7d-99a5-472a-908a-b0fa2b82656f None admin] PATH_INFO: `/v3/projects` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:24.886597 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc4fda7d-99a5-472a-908a-b0fa2b82656f None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:43:24.886906 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc4fda7d-99a5-472a-908a-b0fa2b82656f None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:24.888333 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc4fda7d-99a5-472a-908a-b0fa2b82656f None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:24.891255 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fc5b68d-011c-4145-9ed1-fc862b9cd187 None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:24.900290 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 296/592] 10.222.0.27 () {56 vars in 1148 bytes} [Tue Aug 30 01:43:24 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:43:24.900290 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc4fda7d-99a5-472a-908a-b0fa2b82656f None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:24.935023 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc8d358f-d8b8-4a4f-a8b9-9bfe8af04dba None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:24.936016 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc8d358f-d8b8-4a4f-a8b9-9bfe8af04dba None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:24.936439 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc8d358f-d8b8-4a4f-a8b9-9bfe8af04dba None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:24.936852 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc8d358f-d8b8-4a4f-a8b9-9bfe8af04dba None admin] PATH_INFO: `/v3/projects/service` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:24.937689 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:43:24.945305 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc8d358f-d8b8-4a4f-a8b9-9bfe8af04dba None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:24.946989 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc8d358f-d8b8-4a4f-a8b9-9bfe8af04dba None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:24.959036 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application [None req-dc8d358f-d8b8-4a4f-a8b9-9bfe8af04dba None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 30 01:43:24.959036 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 01:43:24.959036 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 01:43:24.959036 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 01:43:24.959036 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 01:43:24.959036 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 01:43:24.959036 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 01:43:24.959036 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 01:43:24.959036 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 01:43:24.959036 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 01:43:24.959036 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 01:43:24.959036 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 01:43:24.959036 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 30 01:43:24.959036 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 30 01:43:24.959036 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 30 01:43:24.960860 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 30 01:43:24.960860 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 01:43:24.960860 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 01:43:24.960860 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 01:43:24.960860 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 01:43:24.960860 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 01:43:24.960860 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 01:43:24.960860 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 01:43:24.960860 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application with Lock( Aug 30 01:43:24.960860 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 01:43:24.960860 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._enter() Aug 30 01:43:24.960860 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 01:43:24.960860 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 01:43:24.960860 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 01:43:24.960860 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.creator() Aug 30 01:43:24.960860 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 01:43:24.960860 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application created_value = creator( Aug 30 01:43:24.961832 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 30 01:43:24.961832 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 30 01:43:24.961832 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 30 01:43:24.961832 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 30 01:43:24.961832 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 30 01:43:24.961832 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 30 01:43:24.961832 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 30 01:43:24.961832 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application Aug 30 01:43:24.968177 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 297/593] 10.222.0.27 () {56 vars in 1111 bytes} [Tue Aug 30 01:43:24 2022] GET /identity/v3/projects/service => generated 88 bytes in 197 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 01:43:24.969178 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc8d358f-d8b8-4a4f-a8b9-9bfe8af04dba None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:25.005404 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc4fda7d-99a5-472a-908a-b0fa2b82656f None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:25.018216 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:43:25.040397 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e67282d-3e8f-49d2-b890-66357e2de40a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:25.041008 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e67282d-3e8f-49d2-b890-66357e2de40a None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:25.041290 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e67282d-3e8f-49d2-b890-66357e2de40a None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:25.041595 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e67282d-3e8f-49d2-b890-66357e2de40a None admin] PATH_INFO: `/v3/users` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:25.042383 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e67282d-3e8f-49d2-b890-66357e2de40a None admin] RBAC: Adding query filter params (domain_id=default, name=neutron) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:43:25.042702 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e67282d-3e8f-49d2-b890-66357e2de40a None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:25.044132 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e67282d-3e8f-49d2-b890-66357e2de40a None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:25.061992 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 297/594] 10.222.0.27 () {56 vars in 1139 bytes} [Tue Aug 30 01:43:24 2022] GET /identity/v3/users?domain_id=default&name=neutron => generated 374 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:43:25.062788 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e67282d-3e8f-49d2-b890-66357e2de40a None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:25.079583 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc8d358f-d8b8-4a4f-a8b9-9bfe8af04dba None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:25.115020 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3836806f-79b6-47cb-9239-85d1b33526a6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:25.116396 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3836806f-79b6-47cb-9239-85d1b33526a6 None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:25.116822 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3836806f-79b6-47cb-9239-85d1b33526a6 None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:25.117297 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3836806f-79b6-47cb-9239-85d1b33526a6 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:25.128668 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3836806f-79b6-47cb-9239-85d1b33526a6 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:25.130457 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3836806f-79b6-47cb-9239-85d1b33526a6 None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:25.145872 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application [None req-3836806f-79b6-47cb-9239-85d1b33526a6 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 01:43:25.145872 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 01:43:25.145872 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 01:43:25.145872 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 01:43:25.145872 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 01:43:25.145872 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 01:43:25.145872 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 01:43:25.145872 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 01:43:25.145872 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 01:43:25.145872 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 01:43:25.145872 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 01:43:25.145872 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 01:43:25.145872 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 30 01:43:25.145872 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 30 01:43:25.145872 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 30 01:43:25.145872 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 30 01:43:25.147492 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 01:43:25.147492 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 01:43:25.147492 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 01:43:25.147492 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 01:43:25.147492 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 01:43:25.147492 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 01:43:25.147492 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 01:43:25.147492 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application with Lock( Aug 30 01:43:25.147492 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 01:43:25.147492 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._enter() Aug 30 01:43:25.147492 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 01:43:25.147492 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 01:43:25.147492 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 01:43:25.147492 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.creator() Aug 30 01:43:25.147492 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 01:43:25.147492 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application created_value = creator( Aug 30 01:43:25.147492 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 30 01:43:25.148553 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 30 01:43:25.148553 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 01:43:25.148553 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application Aug 30 01:43:25.150153 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 298/595] 10.222.0.27 () {56 vars in 1108 bytes} [Tue Aug 30 01:43:24 2022] GET /identity/v3/domains/Default => generated 87 bytes in 181 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 01:43:25.151059 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3836806f-79b6-47cb-9239-85d1b33526a6 None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:25.160996 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:43:25.170154 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e67282d-3e8f-49d2-b890-66357e2de40a None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:25.206839 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42416660-377f-4ecb-8839-ecd5b2f331fd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:25.207505 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42416660-377f-4ecb-8839-ecd5b2f331fd None admin] REQUEST_METHOD: `PUT` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:25.207790 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42416660-377f-4ecb-8839-ecd5b2f331fd None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:25.208067 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42416660-377f-4ecb-8839-ecd5b2f331fd None admin] PATH_INFO: `/v3/projects/4562f3951fbe465280073e60a11e44e2/users/3caff62093d748f48596574836e12e42/roles/aa59b0c9009a480994d9df81a5f8d404` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:25.222687 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42416660-377f-4ecb-8839-ecd5b2f331fd None admin] RBAC: Authorizing `identity:create_grant(project_id=4562f3951fbe465280073e60a11e44e2, user_id=3caff62093d748f48596574836e12e42, role_id=aa59b0c9009a480994d9df81a5f8d404)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:25.224125 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42416660-377f-4ecb-8839-ecd5b2f331fd None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:25.258546 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:43:25.262329 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3836806f-79b6-47cb-9239-85d1b33526a6 None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:25.296150 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba42ff02-a7ee-4e22-815a-1c9c39904cf3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:25.296753 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba42ff02-a7ee-4e22-815a-1c9c39904cf3 None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:25.297033 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba42ff02-a7ee-4e22-815a-1c9c39904cf3 None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:25.297305 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba42ff02-a7ee-4e22-815a-1c9c39904cf3 None admin] PATH_INFO: `/v3/projects` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:25.298033 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba42ff02-a7ee-4e22-815a-1c9c39904cf3 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81578) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:43:25.298340 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba42ff02-a7ee-4e22-815a-1c9c39904cf3 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:25.309898 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 298/596] 10.222.0.27 () {58 vars in 1439 bytes} [Tue Aug 30 01:43:25 2022] PUT /identity/v3/projects/4562f3951fbe465280073e60a11e44e2/users/3caff62093d748f48596574836e12e42/roles/aa59b0c9009a480994d9df81a5f8d404 => generated 0 bytes in 237 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 01:43:25.310108 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba42ff02-a7ee-4e22-815a-1c9c39904cf3 None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:25.310226 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42416660-377f-4ecb-8839-ecd5b2f331fd None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:25.324457 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 299/597] 10.222.0.27 () {56 vars in 1148 bytes} [Tue Aug 30 01:43:25 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:43:25.325440 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba42ff02-a7ee-4e22-815a-1c9c39904cf3 None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:25.469830 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba42ff02-a7ee-4e22-815a-1c9c39904cf3 None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:25.472164 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42416660-377f-4ecb-8839-ecd5b2f331fd None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:25.509243 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bda6b375-533e-4eb1-8255-036d51f79899 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:25.509892 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bda6b375-533e-4eb1-8255-036d51f79899 None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:25.510252 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bda6b375-533e-4eb1-8255-036d51f79899 None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:25.510568 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bda6b375-533e-4eb1-8255-036d51f79899 None admin] PATH_INFO: `/v3/domains` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:25.511349 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bda6b375-533e-4eb1-8255-036d51f79899 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81578) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:43:25.511710 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bda6b375-533e-4eb1-8255-036d51f79899 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:25.513154 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bda6b375-533e-4eb1-8255-036d51f79899 None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:25.514309 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4ad0f27-95c1-4713-876b-603bdd0522d3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:25.515200 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4ad0f27-95c1-4713-876b-603bdd0522d3 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:25.515660 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4ad0f27-95c1-4713-876b-603bdd0522d3 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:25.516127 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4ad0f27-95c1-4713-876b-603bdd0522d3 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:25.526408 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 300/598] 10.222.0.27 () {56 vars in 1109 bytes} [Tue Aug 30 01:43:25 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 201 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:43:25.527303 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bda6b375-533e-4eb1-8255-036d51f79899 None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:25.536481 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4ad0f27-95c1-4713-876b-603bdd0522d3 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:25.539602 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4ad0f27-95c1-4713-876b-603bdd0522d3 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:25.549852 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:43:25.554875 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:43:25.557266 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application [None req-e4ad0f27-95c1-4713-876b-603bdd0522d3 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 01:43:25.557266 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 01:43:25.557266 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 01:43:25.557266 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 01:43:25.557266 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 01:43:25.557266 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 01:43:25.557266 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 01:43:25.557266 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 01:43:25.557266 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 01:43:25.557266 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 01:43:25.557266 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 01:43:25.557266 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 01:43:25.557266 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 30 01:43:25.557266 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 30 01:43:25.557266 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 30 01:43:25.557266 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 30 01:43:25.564373 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 01:43:25.564373 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 01:43:25.564373 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 01:43:25.564373 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 01:43:25.564373 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 01:43:25.564373 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 01:43:25.564373 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 01:43:25.564373 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application with Lock( Aug 30 01:43:25.564373 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 01:43:25.564373 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._enter() Aug 30 01:43:25.564373 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 01:43:25.564373 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 01:43:25.564373 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 01:43:25.564373 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.creator() Aug 30 01:43:25.564373 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 01:43:25.564373 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application created_value = creator( Aug 30 01:43:25.564373 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 30 01:43:25.565862 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 30 01:43:25.565862 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 01:43:25.565862 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Aug 30 01:43:25.565862 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 299/599] 10.222.0.27 () {56 vars in 1108 bytes} [Tue Aug 30 01:43:25 2022] GET /identity/v3/domains/Default => generated 87 bytes in 259 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 01:43:25.565862 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4ad0f27-95c1-4713-876b-603bdd0522d3 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:25.639878 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bda6b375-533e-4eb1-8255-036d51f79899 None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:25.674912 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4ad0f27-95c1-4713-876b-603bdd0522d3 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:25.675561 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b3a2845-e0e1-459d-8779-6fb814b93255 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:25.676150 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b3a2845-e0e1-459d-8779-6fb814b93255 None admin] REQUEST_METHOD: `PUT` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:25.676425 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b3a2845-e0e1-459d-8779-6fb814b93255 None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:25.676695 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b3a2845-e0e1-459d-8779-6fb814b93255 None admin] PATH_INFO: `/v3/projects/4562f3951fbe465280073e60a11e44e2/users/d64f2e71361644589bec8a06e6eceed1/roles/aa59b0c9009a480994d9df81a5f8d404` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:25.693231 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b3a2845-e0e1-459d-8779-6fb814b93255 None admin] RBAC: Authorizing `identity:create_grant(project_id=4562f3951fbe465280073e60a11e44e2, user_id=d64f2e71361644589bec8a06e6eceed1, role_id=aa59b0c9009a480994d9df81a5f8d404)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:25.695227 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b3a2845-e0e1-459d-8779-6fb814b93255 None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:25.712407 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecd94836-a94e-4caf-8c5e-e595e390dc51 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:25.713051 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecd94836-a94e-4caf-8c5e-e595e390dc51 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:25.713338 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecd94836-a94e-4caf-8c5e-e595e390dc51 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:25.713669 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecd94836-a94e-4caf-8c5e-e595e390dc51 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:25.727377 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ecd94836-a94e-4caf-8c5e-e595e390dc51 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:25.728817 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ecd94836-a94e-4caf-8c5e-e595e390dc51 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:25.740062 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application [None req-ecd94836-a94e-4caf-8c5e-e595e390dc51 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 30 01:43:25.740062 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 01:43:25.740062 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 01:43:25.740062 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 01:43:25.740062 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 01:43:25.740062 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 01:43:25.740062 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 01:43:25.740062 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 01:43:25.740062 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 01:43:25.740062 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 01:43:25.740062 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 01:43:25.740062 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 01:43:25.740062 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 30 01:43:25.740062 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 30 01:43:25.740062 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 30 01:43:25.741973 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 30 01:43:25.741973 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 01:43:25.741973 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 01:43:25.741973 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 01:43:25.741973 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 01:43:25.741973 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 01:43:25.741973 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 01:43:25.741973 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 01:43:25.741973 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application with Lock( Aug 30 01:43:25.741973 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 01:43:25.741973 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._enter() Aug 30 01:43:25.741973 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 01:43:25.741973 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 01:43:25.741973 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 01:43:25.741973 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.creator() Aug 30 01:43:25.741973 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 01:43:25.741973 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application created_value = creator( Aug 30 01:43:25.743002 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 30 01:43:25.743002 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 30 01:43:25.743002 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 30 01:43:25.743002 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 30 01:43:25.743002 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 30 01:43:25.743002 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 30 01:43:25.743002 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 30 01:43:25.743002 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Aug 30 01:43:25.743002 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 300/600] 10.222.0.27 () {56 vars in 1111 bytes} [Tue Aug 30 01:43:25 2022] GET /identity/v3/projects/service => generated 88 bytes in 182 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 01:43:25.743002 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecd94836-a94e-4caf-8c5e-e595e390dc51 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:25.790788 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 301/601] 10.222.0.27 () {58 vars in 1439 bytes} [Tue Aug 30 01:43:25 2022] PUT /identity/v3/projects/4562f3951fbe465280073e60a11e44e2/users/d64f2e71361644589bec8a06e6eceed1/roles/aa59b0c9009a480994d9df81a5f8d404 => generated 0 bytes in 264 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 01:43:25.791993 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b3a2845-e0e1-459d-8779-6fb814b93255 None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:25.804699 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:43:25.883972 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecd94836-a94e-4caf-8c5e-e595e390dc51 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:25.904761 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:43:25.922771 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75d4d041-baf5-40ee-bc21-695d02e36c64 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:25.923400 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75d4d041-baf5-40ee-bc21-695d02e36c64 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:25.923682 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75d4d041-baf5-40ee-bc21-695d02e36c64 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:25.923954 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75d4d041-baf5-40ee-bc21-695d02e36c64 None admin] PATH_INFO: `/v3/domains` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:25.924755 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75d4d041-baf5-40ee-bc21-695d02e36c64 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:43:25.924945 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75d4d041-baf5-40ee-bc21-695d02e36c64 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:25.926518 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75d4d041-baf5-40ee-bc21-695d02e36c64 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:25.945719 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 301/602] 10.222.0.27 () {56 vars in 1109 bytes} [Tue Aug 30 01:43:25 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 200 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:43:25.947208 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-384c5962-a2bf-45b2-ae16-0b37ee105309 None None] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:25.947589 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-384c5962-a2bf-45b2-ae16-0b37ee105309 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:25.947783 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-384c5962-a2bf-45b2-ae16-0b37ee105309 None None] PATH_INFO: `/` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:25.952181 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 302/603] 10.222.0.27 () {54 vars in 927 bytes} [Tue Aug 30 01:43:25 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 01:43:25.953843 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-384c5962-a2bf-45b2-ae16-0b37ee105309 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:25.958839 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b3a2845-e0e1-459d-8779-6fb814b93255 None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:26.004416 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f92f0c2d-02a3-4be6-bb2a-583e26a6d6e2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:26.005027 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f92f0c2d-02a3-4be6-bb2a-583e26a6d6e2 None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:26.005306 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f92f0c2d-02a3-4be6-bb2a-583e26a6d6e2 None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:26.010233 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f92f0c2d-02a3-4be6-bb2a-583e26a6d6e2 None admin] PATH_INFO: `/v3/projects` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:26.010233 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f92f0c2d-02a3-4be6-bb2a-583e26a6d6e2 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81578) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:43:26.010233 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f92f0c2d-02a3-4be6-bb2a-583e26a6d6e2 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:26.010233 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f92f0c2d-02a3-4be6-bb2a-583e26a6d6e2 None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:26.024003 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 302/604] 10.222.0.27 () {56 vars in 1148 bytes} [Tue Aug 30 01:43:25 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 233 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:43:26.025656 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f8049ce-348a-4446-8877-dc243cad5202 None None] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:26.026096 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f8049ce-348a-4446-8877-dc243cad5202 None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:26.026375 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f8049ce-348a-4446-8877-dc243cad5202 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:26.049855 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.auth.core [None req-3f8049ce-348a-4446-8877-dc243cad5202 None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81578) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 01:43:26.051337 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-3f8049ce-348a-4446-8877-dc243cad5202 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:43:26.073314 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-384c5962-a2bf-45b2-ae16-0b37ee105309 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:26.113706 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:43:26.120407 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2d0501d-d45e-4683-b48e-43d30a00c1f3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:26.121278 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2d0501d-d45e-4683-b48e-43d30a00c1f3 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:26.121791 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2d0501d-d45e-4683-b48e-43d30a00c1f3 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:26.122267 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2d0501d-d45e-4683-b48e-43d30a00c1f3 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:26.129424 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2d0501d-d45e-4683-b48e-43d30a00c1f3 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:26.131153 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2d0501d-d45e-4683-b48e-43d30a00c1f3 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:26.142708 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 303/605] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Aug 30 01:43:26 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 118 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 30 01:43:26.143733 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f8049ce-348a-4446-8877-dc243cad5202 None None] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:26.145273 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application [None req-b2d0501d-d45e-4683-b48e-43d30a00c1f3 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 30 01:43:26.145273 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 01:43:26.145273 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 01:43:26.145273 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 01:43:26.145273 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 01:43:26.145273 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 01:43:26.145273 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 01:43:26.145273 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 01:43:26.145273 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 01:43:26.145273 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 01:43:26.145273 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 01:43:26.145273 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 01:43:26.145273 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 30 01:43:26.145273 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 30 01:43:26.145273 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 30 01:43:26.146202 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 30 01:43:26.146202 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 01:43:26.146202 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 01:43:26.146202 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 01:43:26.146202 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 01:43:26.146202 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 01:43:26.146202 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 01:43:26.146202 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 01:43:26.146202 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application with Lock( Aug 30 01:43:26.146202 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 01:43:26.146202 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._enter() Aug 30 01:43:26.146202 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 01:43:26.146202 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 01:43:26.146202 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 01:43:26.146202 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.creator() Aug 30 01:43:26.146202 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 01:43:26.146202 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application created_value = creator( Aug 30 01:43:26.147250 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 30 01:43:26.147250 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 30 01:43:26.147250 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 30 01:43:26.147250 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 30 01:43:26.147250 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 30 01:43:26.147250 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 30 01:43:26.147250 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 30 01:43:26.147250 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Aug 30 01:43:26.150195 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 303/606] 10.222.0.27 () {56 vars in 1111 bytes} [Tue Aug 30 01:43:25 2022] GET /identity/v3/projects/service => generated 88 bytes in 197 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 01:43:26.163183 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2d0501d-d45e-4683-b48e-43d30a00c1f3 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:26.270838 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:43:26.313250 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2d0501d-d45e-4683-b48e-43d30a00c1f3 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:26.357334 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2b0b0a4-a888-4052-bdcd-5e4fdd4773f4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:26.357965 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2b0b0a4-a888-4052-bdcd-5e4fdd4773f4 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:26.358238 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2b0b0a4-a888-4052-bdcd-5e4fdd4773f4 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:26.358504 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2b0b0a4-a888-4052-bdcd-5e4fdd4773f4 None admin] PATH_INFO: `/v3/projects` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:26.359411 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2b0b0a4-a888-4052-bdcd-5e4fdd4773f4 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:43:26.359808 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2b0b0a4-a888-4052-bdcd-5e4fdd4773f4 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:26.361251 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2b0b0a4-a888-4052-bdcd-5e4fdd4773f4 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:26.371877 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f8049ce-348a-4446-8877-dc243cad5202 None None] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:26.381014 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 304/607] 10.222.0.27 () {56 vars in 1148 bytes} [Tue Aug 30 01:43:26 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 219 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:43:26.410538 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2b0b0a4-a888-4052-bdcd-5e4fdd4773f4 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:26.416599 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da259e57-1fe2-4e5e-b34b-3ee39961e294 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:26.417326 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da259e57-1fe2-4e5e-b34b-3ee39961e294 None admin] REQUEST_METHOD: `PUT` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:26.417632 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da259e57-1fe2-4e5e-b34b-3ee39961e294 None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:26.417938 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da259e57-1fe2-4e5e-b34b-3ee39961e294 None admin] PATH_INFO: `/v3/projects/4562f3951fbe465280073e60a11e44e2/users/ab936b3767a8417aa80c8c8f1b84b667/roles/aa59b0c9009a480994d9df81a5f8d404` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:26.435433 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da259e57-1fe2-4e5e-b34b-3ee39961e294 None admin] RBAC: Authorizing `identity:create_grant(project_id=4562f3951fbe465280073e60a11e44e2, user_id=ab936b3767a8417aa80c8c8f1b84b667, role_id=aa59b0c9009a480994d9df81a5f8d404)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:26.436864 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da259e57-1fe2-4e5e-b34b-3ee39961e294 None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:26.453710 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:43:26.527935 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2b0b0a4-a888-4052-bdcd-5e4fdd4773f4 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:26.545494 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 304/608] 10.222.0.27 () {58 vars in 1439 bytes} [Tue Aug 30 01:43:26 2022] PUT /identity/v3/projects/4562f3951fbe465280073e60a11e44e2/users/ab936b3767a8417aa80c8c8f1b84b667/roles/aa59b0c9009a480994d9df81a5f8d404 => generated 0 bytes in 402 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 01:43:26.549606 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da259e57-1fe2-4e5e-b34b-3ee39961e294 None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:26.607551 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d46a0007-8d0c-4279-88b4-94ae38821479 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:26.608467 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d46a0007-8d0c-4279-88b4-94ae38821479 None admin] REQUEST_METHOD: `PUT` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:26.608911 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d46a0007-8d0c-4279-88b4-94ae38821479 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:26.609326 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d46a0007-8d0c-4279-88b4-94ae38821479 None admin] PATH_INFO: `/v3/projects/4562f3951fbe465280073e60a11e44e2/users/f3ea1580c56f4814a92bbb2acf542905/roles/aa59b0c9009a480994d9df81a5f8d404` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:26.641878 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d46a0007-8d0c-4279-88b4-94ae38821479 None admin] RBAC: Authorizing `identity:create_grant(project_id=4562f3951fbe465280073e60a11e44e2, user_id=f3ea1580c56f4814a92bbb2acf542905, role_id=aa59b0c9009a480994d9df81a5f8d404)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:26.643634 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d46a0007-8d0c-4279-88b4-94ae38821479 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:26.711434 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da259e57-1fe2-4e5e-b34b-3ee39961e294 None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:26.725824 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 305/609] 10.222.0.27 () {58 vars in 1439 bytes} [Tue Aug 30 01:43:26 2022] PUT /identity/v3/projects/4562f3951fbe465280073e60a11e44e2/users/f3ea1580c56f4814a92bbb2acf542905/roles/aa59b0c9009a480994d9df81a5f8d404 => generated 0 bytes in 312 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 01:43:26.736189 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:43:26.765053 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96ddabe6-6caf-458f-9d79-3e1089a210eb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:26.765907 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96ddabe6-6caf-458f-9d79-3e1089a210eb None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:26.766193 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96ddabe6-6caf-458f-9d79-3e1089a210eb None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:26.766467 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96ddabe6-6caf-458f-9d79-3e1089a210eb None admin] PATH_INFO: `/v3/roles/service` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:26.783253 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96ddabe6-6caf-458f-9d79-3e1089a210eb None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 30 01:43:26.788833 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application [None req-96ddabe6-6caf-458f-9d79-3e1089a210eb None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Aug 30 01:43:26.788833 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 01:43:26.788833 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 01:43:26.788833 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 01:43:26.788833 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 01:43:26.788833 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 01:43:26.788833 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 01:43:26.788833 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 01:43:26.788833 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 01:43:26.788833 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 01:43:26.788833 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 01:43:26.788833 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 01:43:26.788833 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 30 01:43:26.788833 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 30 01:43:26.788833 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 30 01:43:26.788833 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 30 01:43:26.789999 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 30 01:43:26.789999 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 30 01:43:26.789999 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 30 01:43:26.789999 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 30 01:43:26.789999 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 01:43:26.789999 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 01:43:26.789999 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 01:43:26.789999 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 01:43:26.789999 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 01:43:26.789999 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 01:43:26.789999 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 01:43:26.789999 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application with Lock( Aug 30 01:43:26.789999 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 01:43:26.789999 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._enter() Aug 30 01:43:26.789999 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 01:43:26.789999 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 01:43:26.791035 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 01:43:26.791035 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.creator() Aug 30 01:43:26.791035 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 01:43:26.791035 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application created_value = creator( Aug 30 01:43:26.791035 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 30 01:43:26.791035 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 30 01:43:26.791035 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 30 01:43:26.791035 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 30 01:43:26.791035 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 30 01:43:26.791035 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 30 01:43:26.791035 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Aug 30 01:43:26.791035 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application Aug 30 01:43:26.791035 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 305/610] 10.222.0.27 () {56 vars in 1102 bytes} [Tue Aug 30 01:43:26 2022] GET /identity/v3/roles/service => generated 85 bytes in 244 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 01:43:26.797966 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d46a0007-8d0c-4279-88b4-94ae38821479 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:26.915948 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:43:26.942488 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d46a0007-8d0c-4279-88b4-94ae38821479 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:26.992846 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7db0f5d-23f3-4983-bd2d-035b85a37974 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:26.993488 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7db0f5d-23f3-4983-bd2d-035b85a37974 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:26.993758 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7db0f5d-23f3-4983-bd2d-035b85a37974 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:26.994031 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7db0f5d-23f3-4983-bd2d-035b85a37974 None admin] PATH_INFO: `/v3/roles` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:26.994710 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7db0f5d-23f3-4983-bd2d-035b85a37974 None admin] RBAC: Adding query filter params (name=service) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:43:26.995014 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7db0f5d-23f3-4983-bd2d-035b85a37974 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:26.996381 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7db0f5d-23f3-4983-bd2d-035b85a37974 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:27.005558 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 306/611] 10.222.0.27 () {56 vars in 1103 bytes} [Tue Aug 30 01:43:26 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 208 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 30 01:43:27.011729 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96ddabe6-6caf-458f-9d79-3e1089a210eb None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:27.122893 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96ddabe6-6caf-458f-9d79-3e1089a210eb None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:27.169731 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52b52602-52e6-476e-9486-ae65b9750e98 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:27.170302 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52b52602-52e6-476e-9486-ae65b9750e98 None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:27.170581 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52b52602-52e6-476e-9486-ae65b9750e98 None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:27.170899 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52b52602-52e6-476e-9486-ae65b9750e98 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:27.188935 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52b52602-52e6-476e-9486-ae65b9750e98 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:27.191222 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52b52602-52e6-476e-9486-ae65b9750e98 None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:27.218547 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application [None req-52b52602-52e6-476e-9486-ae65b9750e98 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 01:43:27.218547 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 01:43:27.218547 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 01:43:27.218547 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 01:43:27.218547 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 01:43:27.218547 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 01:43:27.218547 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 01:43:27.218547 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 01:43:27.218547 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 01:43:27.218547 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 01:43:27.218547 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 01:43:27.218547 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 01:43:27.218547 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 30 01:43:27.218547 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 30 01:43:27.218547 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 30 01:43:27.218547 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 30 01:43:27.219550 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 01:43:27.219550 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 01:43:27.219550 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 01:43:27.219550 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 01:43:27.219550 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 01:43:27.219550 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 01:43:27.219550 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 01:43:27.219550 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application with Lock( Aug 30 01:43:27.219550 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 01:43:27.219550 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._enter() Aug 30 01:43:27.219550 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 01:43:27.219550 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 01:43:27.219550 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 01:43:27.219550 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.creator() Aug 30 01:43:27.219550 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 01:43:27.219550 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application created_value = creator( Aug 30 01:43:27.219550 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 30 01:43:27.220436 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 30 01:43:27.220436 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 01:43:27.220436 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application Aug 30 01:43:27.222262 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 306/612] 10.222.0.27 () {56 vars in 1108 bytes} [Tue Aug 30 01:43:27 2022] GET /identity/v3/domains/Default => generated 87 bytes in 211 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 01:43:27.241100 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7db0f5d-23f3-4983-bd2d-035b85a37974 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:27.377906 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7db0f5d-23f3-4983-bd2d-035b85a37974 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:27.416204 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdbb2830-9c1d-4ea6-a6f7-07eb9c8fbee8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:27.416822 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdbb2830-9c1d-4ea6-a6f7-07eb9c8fbee8 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:27.417107 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdbb2830-9c1d-4ea6-a6f7-07eb9c8fbee8 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:27.417384 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdbb2830-9c1d-4ea6-a6f7-07eb9c8fbee8 None admin] PATH_INFO: `/v3/domains` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:27.418067 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fdbb2830-9c1d-4ea6-a6f7-07eb9c8fbee8 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:43:27.418373 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fdbb2830-9c1d-4ea6-a6f7-07eb9c8fbee8 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:27.419806 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fdbb2830-9c1d-4ea6-a6f7-07eb9c8fbee8 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:27.432829 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 307/613] 10.222.0.27 () {56 vars in 1109 bytes} [Tue Aug 30 01:43:27 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 193 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:43:27.438218 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52b52602-52e6-476e-9486-ae65b9750e98 None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:27.478556 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:43:27.480011 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7368d6fa-3bd1-4dbe-958b-2f580a7cb9c8 None None] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:27.480306 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7368d6fa-3bd1-4dbe-958b-2f580a7cb9c8 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:27.480587 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7368d6fa-3bd1-4dbe-958b-2f580a7cb9c8 None None] PATH_INFO: `/` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:27.482622 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 308/614] 10.222.0.27 () {54 vars in 927 bytes} [Tue Aug 30 01:43:27 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 01:43:27.487333 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a768407-12ce-49fb-8928-d2e0b57885c9 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:27.487616 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a768407-12ce-49fb-8928-d2e0b57885c9 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:27.487892 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a768407-12ce-49fb-8928-d2e0b57885c9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:27.501298 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:43:27.512212 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-9a768407-12ce-49fb-8928-d2e0b57885c9 None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 01:43:27.515419 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-9a768407-12ce-49fb-8928-d2e0b57885c9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:43:27.623576 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52b52602-52e6-476e-9486-ae65b9750e98 None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:27.632744 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 309/615] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Aug 30 01:43:27 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 146 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Aug 30 01:43:27.682005 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ea3e567-d4c1-4a5a-9d6f-b96423ade349 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:27.682552 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ea3e567-d4c1-4a5a-9d6f-b96423ade349 None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:27.682835 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ea3e567-d4c1-4a5a-9d6f-b96423ade349 None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:27.683101 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ea3e567-d4c1-4a5a-9d6f-b96423ade349 None admin] PATH_INFO: `/v3/users/swift` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:27.692367 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ea3e567-d4c1-4a5a-9d6f-b96423ade349 None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:27.694359 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ea3e567-d4c1-4a5a-9d6f-b96423ade349 None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:27.710652 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application [None req-6ea3e567-d4c1-4a5a-9d6f-b96423ade349 None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Aug 30 01:43:27.710652 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 01:43:27.710652 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 01:43:27.710652 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 01:43:27.710652 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 01:43:27.710652 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 01:43:27.710652 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 01:43:27.710652 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 01:43:27.710652 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 01:43:27.710652 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 01:43:27.710652 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 01:43:27.710652 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 01:43:27.710652 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 30 01:43:27.710652 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 30 01:43:27.710652 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 30 01:43:27.710652 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 30 01:43:27.711620 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 01:43:27.711620 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 01:43:27.711620 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 30 01:43:27.711620 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 30 01:43:27.711620 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 30 01:43:27.711620 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 30 01:43:27.711620 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 01:43:27.711620 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 01:43:27.711620 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 01:43:27.711620 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 01:43:27.711620 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 01:43:27.711620 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application with Lock( Aug 30 01:43:27.711620 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 01:43:27.711620 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._enter() Aug 30 01:43:27.711620 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 01:43:27.711620 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 01:43:27.711620 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 01:43:27.711620 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.creator() Aug 30 01:43:27.712536 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 01:43:27.712536 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application created_value = creator( Aug 30 01:43:27.712536 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 30 01:43:27.712536 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 30 01:43:27.712536 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 30 01:43:27.712536 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 30 01:43:27.712536 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 30 01:43:27.712536 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 30 01:43:27.712536 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Aug 30 01:43:27.712536 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application Aug 30 01:43:27.712536 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 307/616] 10.222.0.27 () {56 vars in 1096 bytes} [Tue Aug 30 01:43:27 2022] GET /identity/v3/users/swift => generated 83 bytes in 274 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 01:43:27.719871 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a768407-12ce-49fb-8928-d2e0b57885c9 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:27.861339 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a768407-12ce-49fb-8928-d2e0b57885c9 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:27.903162 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d6c8efd-9d35-4761-b427-4e5783eab51c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:27.903769 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d6c8efd-9d35-4761-b427-4e5783eab51c None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:27.904043 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d6c8efd-9d35-4761-b427-4e5783eab51c None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:27.904311 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d6c8efd-9d35-4761-b427-4e5783eab51c None admin] PATH_INFO: `/v3/users` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:27.905100 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d6c8efd-9d35-4761-b427-4e5783eab51c None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:43:27.905405 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d6c8efd-9d35-4761-b427-4e5783eab51c None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:27.906842 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d6c8efd-9d35-4761-b427-4e5783eab51c None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:27.909555 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:43:27.932833 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 310/617] 10.222.0.27 () {56 vars in 1135 bytes} [Tue Aug 30 01:43:27 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 213 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:43:27.938291 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ea3e567-d4c1-4a5a-9d6f-b96423ade349 None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:28.041859 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:43:28.059416 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ea3e567-d4c1-4a5a-9d6f-b96423ade349 None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:28.082922 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d6c8efd-9d35-4761-b427-4e5783eab51c None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:28.134168 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11299943-db5f-45d3-af2c-e20d41c8eeda None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:28.134811 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11299943-db5f-45d3-af2c-e20d41c8eeda None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:28.135088 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11299943-db5f-45d3-af2c-e20d41c8eeda None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:28.135364 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11299943-db5f-45d3-af2c-e20d41c8eeda None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:28.157629 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11299943-db5f-45d3-af2c-e20d41c8eeda None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:28.160744 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11299943-db5f-45d3-af2c-e20d41c8eeda None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:28.184749 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application [None req-11299943-db5f-45d3-af2c-e20d41c8eeda None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 01:43:28.184749 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 01:43:28.184749 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 01:43:28.184749 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 01:43:28.184749 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 01:43:28.184749 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 01:43:28.184749 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 01:43:28.184749 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 01:43:28.184749 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 01:43:28.184749 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 01:43:28.184749 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 01:43:28.184749 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 01:43:28.184749 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 30 01:43:28.184749 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 30 01:43:28.184749 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 30 01:43:28.184749 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 30 01:43:28.185989 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 01:43:28.185989 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 01:43:28.185989 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 01:43:28.185989 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 01:43:28.185989 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 01:43:28.185989 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 01:43:28.185989 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 01:43:28.185989 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application with Lock( Aug 30 01:43:28.185989 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 01:43:28.185989 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._enter() Aug 30 01:43:28.185989 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 01:43:28.185989 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 01:43:28.185989 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 01:43:28.185989 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.creator() Aug 30 01:43:28.185989 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 01:43:28.185989 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application created_value = creator( Aug 30 01:43:28.185989 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 30 01:43:28.187054 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 30 01:43:28.187054 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 01:43:28.187054 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application Aug 30 01:43:28.189237 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 308/618] 10.222.0.27 () {56 vars in 1108 bytes} [Tue Aug 30 01:43:27 2022] GET /identity/v3/domains/Default => generated 87 bytes in 251 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 01:43:28.200244 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11299943-db5f-45d3-af2c-e20d41c8eeda None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:28.262068 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d6c8efd-9d35-4761-b427-4e5783eab51c None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:28.315780 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c849e2ce-8a80-4e63-9716-8f6af7cd98c0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:28.316418 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c849e2ce-8a80-4e63-9716-8f6af7cd98c0 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:28.316699 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c849e2ce-8a80-4e63-9716-8f6af7cd98c0 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:28.317384 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c849e2ce-8a80-4e63-9716-8f6af7cd98c0 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:28.332475 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c849e2ce-8a80-4e63-9716-8f6af7cd98c0 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 30 01:43:28.339373 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application [None req-c849e2ce-8a80-4e63-9716-8f6af7cd98c0 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Aug 30 01:43:28.339373 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 01:43:28.339373 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 01:43:28.339373 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 01:43:28.339373 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 01:43:28.339373 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 01:43:28.339373 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 01:43:28.339373 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 01:43:28.339373 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 01:43:28.339373 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 01:43:28.339373 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 01:43:28.339373 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 01:43:28.339373 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 30 01:43:28.339373 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 30 01:43:28.339373 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 30 01:43:28.339373 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 30 01:43:28.340330 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 30 01:43:28.340330 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 30 01:43:28.340330 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 30 01:43:28.340330 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 30 01:43:28.340330 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 01:43:28.340330 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 01:43:28.340330 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 01:43:28.340330 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 01:43:28.340330 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 01:43:28.340330 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 01:43:28.340330 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 01:43:28.340330 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application with Lock( Aug 30 01:43:28.340330 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 01:43:28.340330 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._enter() Aug 30 01:43:28.340330 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 01:43:28.340330 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 01:43:28.341208 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 01:43:28.341208 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.creator() Aug 30 01:43:28.341208 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 01:43:28.341208 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application created_value = creator( Aug 30 01:43:28.341208 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 30 01:43:28.341208 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 30 01:43:28.341208 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 30 01:43:28.341208 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 30 01:43:28.341208 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 30 01:43:28.341208 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 30 01:43:28.341208 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Aug 30 01:43:28.341208 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Aug 30 01:43:28.342460 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 311/619] 10.222.0.27 () {56 vars in 1102 bytes} [Tue Aug 30 01:43:28 2022] GET /identity/v3/roles/service => generated 85 bytes in 260 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 30 01:43:28.343589 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f6e2161-db01-48aa-95f5-277ae18804cc None None] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:28.343589 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f6e2161-db01-48aa-95f5-277ae18804cc None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:28.343754 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f6e2161-db01-48aa-95f5-277ae18804cc None None] PATH_INFO: `/` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:28.349574 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 312/620] 10.222.0.27 () {54 vars in 927 bytes} [Tue Aug 30 01:43:28 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 01:43:28.350998 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f6e2161-db01-48aa-95f5-277ae18804cc None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:28.383367 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11299943-db5f-45d3-af2c-e20d41c8eeda None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:28.423580 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:43:28.429945 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:43:28.430945 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92507326-4a02-4980-8ac2-2992a4036932 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:28.431783 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92507326-4a02-4980-8ac2-2992a4036932 None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:28.432161 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92507326-4a02-4980-8ac2-2992a4036932 None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:28.454145 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92507326-4a02-4980-8ac2-2992a4036932 None admin] PATH_INFO: `/v3/domains` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:28.454145 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92507326-4a02-4980-8ac2-2992a4036932 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81578) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:43:28.454145 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92507326-4a02-4980-8ac2-2992a4036932 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:28.454145 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92507326-4a02-4980-8ac2-2992a4036932 None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:28.454145 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 309/621] 10.222.0.27 () {56 vars in 1109 bytes} [Tue Aug 30 01:43:28 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 249 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:43:28.454145 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7eada301-c7be-40ef-ab7f-70f811343291 None None] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:28.455090 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7eada301-c7be-40ef-ab7f-70f811343291 None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:28.455090 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7eada301-c7be-40ef-ab7f-70f811343291 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:28.472461 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.auth.core [None req-7eada301-c7be-40ef-ab7f-70f811343291 None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81578) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 01:43:28.473885 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-7eada301-c7be-40ef-ab7f-70f811343291 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:43:28.489958 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f6e2161-db01-48aa-95f5-277ae18804cc None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:28.529780 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c2d3314-a22a-4ba4-bb47-ac85b3238202 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:28.530637 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c2d3314-a22a-4ba4-bb47-ac85b3238202 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:28.531060 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c2d3314-a22a-4ba4-bb47-ac85b3238202 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:28.531647 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c2d3314-a22a-4ba4-bb47-ac85b3238202 None admin] PATH_INFO: `/v3/roles` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:28.532600 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c2d3314-a22a-4ba4-bb47-ac85b3238202 None admin] RBAC: Adding query filter params (name=service) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:43:28.533047 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c2d3314-a22a-4ba4-bb47-ac85b3238202 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:28.534607 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c2d3314-a22a-4ba4-bb47-ac85b3238202 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:28.545319 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 313/622] 10.222.0.27 () {56 vars in 1103 bytes} [Tue Aug 30 01:43:28 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 195 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:43:28.546643 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c2d3314-a22a-4ba4-bb47-ac85b3238202 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:28.567273 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 310/623] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Aug 30 01:43:28 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 118 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 30 01:43:28.568356 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7eada301-c7be-40ef-ab7f-70f811343291 None None] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:28.655667 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c2d3314-a22a-4ba4-bb47-ac85b3238202 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:28.673878 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:43:28.707675 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7eada301-c7be-40ef-ab7f-70f811343291 None None] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:28.722081 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ade840ea-0356-40b4-aec0-69c9f73324c8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:28.722930 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ade840ea-0356-40b4-aec0-69c9f73324c8 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:28.723393 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ade840ea-0356-40b4-aec0-69c9f73324c8 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:28.723922 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ade840ea-0356-40b4-aec0-69c9f73324c8 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:28.729868 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:43:28.737118 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ade840ea-0356-40b4-aec0-69c9f73324c8 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:28.745523 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31e21f66-0d6d-4455-ad92-31c9aabb4316 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:28.745523 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31e21f66-0d6d-4455-ad92-31c9aabb4316 None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:28.745523 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31e21f66-0d6d-4455-ad92-31c9aabb4316 None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:28.745523 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31e21f66-0d6d-4455-ad92-31c9aabb4316 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:28.746894 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ade840ea-0356-40b4-aec0-69c9f73324c8 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:28.752183 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31e21f66-0d6d-4455-ad92-31c9aabb4316 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:28.753690 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31e21f66-0d6d-4455-ad92-31c9aabb4316 None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:28.765759 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application [None req-ade840ea-0356-40b4-aec0-69c9f73324c8 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 30 01:43:28.765759 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 01:43:28.765759 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 01:43:28.765759 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 01:43:28.765759 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 01:43:28.765759 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 01:43:28.765759 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 01:43:28.765759 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 01:43:28.765759 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 01:43:28.765759 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 01:43:28.765759 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 01:43:28.765759 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 01:43:28.765759 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 30 01:43:28.765759 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 30 01:43:28.765759 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 30 01:43:28.766740 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 30 01:43:28.766740 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 01:43:28.766740 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 01:43:28.766740 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 01:43:28.766740 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 01:43:28.766740 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 01:43:28.766740 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 01:43:28.766740 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 01:43:28.766740 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application with Lock( Aug 30 01:43:28.766740 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 01:43:28.766740 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._enter() Aug 30 01:43:28.766740 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 01:43:28.766740 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 01:43:28.766740 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 01:43:28.766740 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.creator() Aug 30 01:43:28.766740 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 01:43:28.766740 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application created_value = creator( Aug 30 01:43:28.767625 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 30 01:43:28.767625 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 30 01:43:28.767625 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 30 01:43:28.767625 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 30 01:43:28.767625 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 30 01:43:28.767625 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 30 01:43:28.767625 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 30 01:43:28.767625 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Aug 30 01:43:28.769444 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application [None req-31e21f66-0d6d-4455-ad92-31c9aabb4316 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 01:43:28.769444 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 01:43:28.769444 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 01:43:28.769444 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 01:43:28.769444 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 01:43:28.769444 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 01:43:28.769444 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 01:43:28.769444 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 01:43:28.769444 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 01:43:28.769444 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 01:43:28.769444 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 01:43:28.769444 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 01:43:28.769444 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 30 01:43:28.769444 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 30 01:43:28.769444 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 30 01:43:28.769444 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 30 01:43:28.770380 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 01:43:28.770380 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 01:43:28.770380 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 01:43:28.770380 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 01:43:28.770380 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 01:43:28.770380 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 01:43:28.770380 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 01:43:28.770380 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application with Lock( Aug 30 01:43:28.770380 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 01:43:28.770380 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._enter() Aug 30 01:43:28.770380 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 01:43:28.770380 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 01:43:28.770380 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 01:43:28.770380 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.creator() Aug 30 01:43:28.770380 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 01:43:28.770380 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application created_value = creator( Aug 30 01:43:28.770380 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 30 01:43:28.771269 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 30 01:43:28.771269 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 01:43:28.771269 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application Aug 30 01:43:28.772566 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 314/624] 10.222.0.27 () {56 vars in 1111 bytes} [Tue Aug 30 01:43:28 2022] GET /identity/v3/projects/service => generated 88 bytes in 226 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 01:43:28.778316 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ade840ea-0356-40b4-aec0-69c9f73324c8 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:28.779050 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 311/625] 10.222.0.27 () {56 vars in 1108 bytes} [Tue Aug 30 01:43:28 2022] GET /identity/v3/domains/Default => generated 87 bytes in 211 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 01:43:28.784992 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31e21f66-0d6d-4455-ad92-31c9aabb4316 None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:28.899217 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ade840ea-0356-40b4-aec0-69c9f73324c8 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:28.948143 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75f97d8c-968e-48dd-80af-86dbfe70f741 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:28.948904 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75f97d8c-968e-48dd-80af-86dbfe70f741 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:28.949282 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75f97d8c-968e-48dd-80af-86dbfe70f741 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:28.949615 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75f97d8c-968e-48dd-80af-86dbfe70f741 None admin] PATH_INFO: `/v3/projects` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:28.950371 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75f97d8c-968e-48dd-80af-86dbfe70f741 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:43:28.950680 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75f97d8c-968e-48dd-80af-86dbfe70f741 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:28.952305 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75f97d8c-968e-48dd-80af-86dbfe70f741 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:28.958580 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31e21f66-0d6d-4455-ad92-31c9aabb4316 None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:28.965802 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 315/626] 10.222.0.27 () {56 vars in 1148 bytes} [Tue Aug 30 01:43:28 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 188 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 30 01:43:28.966567 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75f97d8c-968e-48dd-80af-86dbfe70f741 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:28.999364 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ced13d6f-3dcb-4e10-9214-a14848b0e302 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:28.999974 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ced13d6f-3dcb-4e10-9214-a14848b0e302 None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:29.000255 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ced13d6f-3dcb-4e10-9214-a14848b0e302 None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:29.000523 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ced13d6f-3dcb-4e10-9214-a14848b0e302 None admin] PATH_INFO: `/v3/domains` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:29.001168 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ced13d6f-3dcb-4e10-9214-a14848b0e302 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81578) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:43:29.001629 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ced13d6f-3dcb-4e10-9214-a14848b0e302 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:29.002908 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:43:29.003658 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ced13d6f-3dcb-4e10-9214-a14848b0e302 None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:29.016514 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 312/627] 10.222.0.27 () {56 vars in 1109 bytes} [Tue Aug 30 01:43:28 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 232 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:43:29.017762 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78c6c8c4-734c-4dfd-b4a5-e3080b53f0e2 None None] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:29.024610 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78c6c8c4-734c-4dfd-b4a5-e3080b53f0e2 None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:29.024610 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78c6c8c4-734c-4dfd-b4a5-e3080b53f0e2 None None] PATH_INFO: `/` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:29.024610 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 313/628] 10.222.0.27 () {54 vars in 927 bytes} [Tue Aug 30 01:43:29 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 01:43:29.024610 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78c6c8c4-734c-4dfd-b4a5-e3080b53f0e2 None None] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:29.058602 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:43:29.087853 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75f97d8c-968e-48dd-80af-86dbfe70f741 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:29.125239 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-297b4f23-5754-4887-98ae-05203653a308 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:29.126081 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-297b4f23-5754-4887-98ae-05203653a308 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:29.126255 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-297b4f23-5754-4887-98ae-05203653a308 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:29.126527 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-297b4f23-5754-4887-98ae-05203653a308 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:29.131016 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78c6c8c4-734c-4dfd-b4a5-e3080b53f0e2 None None] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:29.138338 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-297b4f23-5754-4887-98ae-05203653a308 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 30 01:43:29.143123 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application [None req-297b4f23-5754-4887-98ae-05203653a308 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Aug 30 01:43:29.143123 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 01:43:29.143123 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 01:43:29.143123 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 01:43:29.143123 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 01:43:29.143123 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 01:43:29.143123 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 01:43:29.143123 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 01:43:29.143123 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 01:43:29.143123 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 01:43:29.143123 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 01:43:29.143123 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 01:43:29.143123 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 30 01:43:29.143123 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 30 01:43:29.143123 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 30 01:43:29.143123 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 30 01:43:29.144133 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 30 01:43:29.144133 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 30 01:43:29.144133 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 30 01:43:29.144133 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 30 01:43:29.144133 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 01:43:29.144133 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 01:43:29.144133 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 01:43:29.144133 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 01:43:29.144133 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 01:43:29.144133 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 01:43:29.144133 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 01:43:29.144133 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application with Lock( Aug 30 01:43:29.144133 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 01:43:29.144133 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._enter() Aug 30 01:43:29.144133 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 01:43:29.144133 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 01:43:29.144975 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 01:43:29.144975 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.creator() Aug 30 01:43:29.144975 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 01:43:29.144975 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application created_value = creator( Aug 30 01:43:29.144975 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 30 01:43:29.144975 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 30 01:43:29.144975 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 30 01:43:29.144975 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 30 01:43:29.144975 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 30 01:43:29.144975 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 30 01:43:29.144975 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Aug 30 01:43:29.144975 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Aug 30 01:43:29.144975 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 316/629] 10.222.0.27 () {56 vars in 1102 bytes} [Tue Aug 30 01:43:28 2022] GET /identity/v3/roles/service => generated 85 bytes in 178 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 01:43:29.144975 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-297b4f23-5754-4887-98ae-05203653a308 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:29.175228 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c244b8aa-1e40-4df1-8501-f0d04af7f3e4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:29.175904 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c244b8aa-1e40-4df1-8501-f0d04af7f3e4 None admin] REQUEST_METHOD: `PUT` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:29.176181 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c244b8aa-1e40-4df1-8501-f0d04af7f3e4 None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:29.176454 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c244b8aa-1e40-4df1-8501-f0d04af7f3e4 None admin] PATH_INFO: `/v3/projects/4562f3951fbe465280073e60a11e44e2/users/b4f0008fde424f22bc1f7307cf422774/roles/aa59b0c9009a480994d9df81a5f8d404` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:29.192715 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c244b8aa-1e40-4df1-8501-f0d04af7f3e4 None admin] RBAC: Authorizing `identity:create_grant(project_id=4562f3951fbe465280073e60a11e44e2, user_id=b4f0008fde424f22bc1f7307cf422774, role_id=aa59b0c9009a480994d9df81a5f8d404)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:29.194517 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c244b8aa-1e40-4df1-8501-f0d04af7f3e4 None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:29.246713 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:43:29.252816 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-297b4f23-5754-4887-98ae-05203653a308 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:29.281792 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 314/630] 10.222.0.27 () {58 vars in 1439 bytes} [Tue Aug 30 01:43:29 2022] PUT /identity/v3/projects/4562f3951fbe465280073e60a11e44e2/users/b4f0008fde424f22bc1f7307cf422774/roles/aa59b0c9009a480994d9df81a5f8d404 => generated 0 bytes in 251 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 01:43:29.283403 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5928cfa1-e0a5-4a14-8992-d14042c176da None None] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:29.283694 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5928cfa1-e0a5-4a14-8992-d14042c176da None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:29.283968 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5928cfa1-e0a5-4a14-8992-d14042c176da None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:29.295343 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddf57fe6-93ac-4651-a72c-cf577f63d703 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:29.295991 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddf57fe6-93ac-4651-a72c-cf577f63d703 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:29.296271 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddf57fe6-93ac-4651-a72c-cf577f63d703 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:29.296546 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddf57fe6-93ac-4651-a72c-cf577f63d703 None admin] PATH_INFO: `/v3/users/glance` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:29.304742 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ddf57fe6-93ac-4651-a72c-cf577f63d703 None admin] RBAC: Authorizing `identity:get_user(user_id=glance)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:29.306237 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ddf57fe6-93ac-4651-a72c-cf577f63d703 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:29.315372 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.auth.core [None req-5928cfa1-e0a5-4a14-8992-d14042c176da None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81578) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 01:43:29.317088 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-5928cfa1-e0a5-4a14-8992-d14042c176da None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:43:29.324661 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application [None req-ddf57fe6-93ac-4651-a72c-cf577f63d703 None admin] Could not find user: glance.: keystone.exception.UserNotFound: Could not find user: glance. Aug 30 01:43:29.324661 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 01:43:29.324661 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 01:43:29.324661 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 01:43:29.324661 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 01:43:29.324661 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 01:43:29.324661 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 01:43:29.324661 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 01:43:29.324661 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 01:43:29.324661 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 01:43:29.324661 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 01:43:29.324661 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 01:43:29.324661 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 30 01:43:29.324661 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 30 01:43:29.324661 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 30 01:43:29.324661 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 30 01:43:29.325726 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 01:43:29.325726 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 01:43:29.325726 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 30 01:43:29.325726 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 30 01:43:29.325726 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 30 01:43:29.325726 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 30 01:43:29.325726 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 01:43:29.325726 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 01:43:29.325726 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 01:43:29.325726 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 01:43:29.325726 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 01:43:29.325726 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application with Lock( Aug 30 01:43:29.325726 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 01:43:29.325726 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._enter() Aug 30 01:43:29.325726 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 01:43:29.325726 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 01:43:29.325726 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 01:43:29.325726 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.creator() Aug 30 01:43:29.326678 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 01:43:29.326678 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application created_value = creator( Aug 30 01:43:29.326678 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 30 01:43:29.326678 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 30 01:43:29.326678 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 30 01:43:29.326678 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 30 01:43:29.326678 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 30 01:43:29.326678 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 30 01:43:29.326678 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance. Aug 30 01:43:29.326678 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Aug 30 01:43:29.326678 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 317/631] 10.222.0.27 () {56 vars in 1099 bytes} [Tue Aug 30 01:43:29 2022] GET /identity/v3/users/glance => generated 84 bytes in 181 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 01:43:29.327283 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddf57fe6-93ac-4651-a72c-cf577f63d703 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:29.429880 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 315/632] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Aug 30 01:43:29 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 147 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 30 01:43:29.430916 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5928cfa1-e0a5-4a14-8992-d14042c176da None None] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:29.447507 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddf57fe6-93ac-4651-a72c-cf577f63d703 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:29.453313 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:43:29.495680 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7519712b-96b8-489b-a7c5-bb52b57ea365 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:29.496299 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7519712b-96b8-489b-a7c5-bb52b57ea365 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:29.496583 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7519712b-96b8-489b-a7c5-bb52b57ea365 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:29.496913 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7519712b-96b8-489b-a7c5-bb52b57ea365 None admin] PATH_INFO: `/v3/roles` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:29.497569 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7519712b-96b8-489b-a7c5-bb52b57ea365 None admin] RBAC: Adding query filter params (name=service) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:43:29.497880 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7519712b-96b8-489b-a7c5-bb52b57ea365 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:29.499269 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7519712b-96b8-489b-a7c5-bb52b57ea365 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:29.509721 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 318/633] 10.222.0.27 () {56 vars in 1103 bytes} [Tue Aug 30 01:43:29 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 183 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:43:29.511024 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64d33ab0-5b7f-43d5-add0-819e704181c7 None None] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:29.511317 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64d33ab0-5b7f-43d5-add0-819e704181c7 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:29.511596 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64d33ab0-5b7f-43d5-add0-819e704181c7 None None] PATH_INFO: `/` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:29.513036 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 319/634] 10.222.0.27 () {54 vars in 927 bytes} [Tue Aug 30 01:43:29 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 01:43:29.517885 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64d33ab0-5b7f-43d5-add0-819e704181c7 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:29.544412 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:43:29.555402 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5928cfa1-e0a5-4a14-8992-d14042c176da None None] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:29.589779 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efdafad4-f4a2-4915-8ce6-145762a0a26f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:29.590495 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efdafad4-f4a2-4915-8ce6-145762a0a26f None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:29.590844 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efdafad4-f4a2-4915-8ce6-145762a0a26f None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:29.591215 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efdafad4-f4a2-4915-8ce6-145762a0a26f None admin] PATH_INFO: `/v3/users` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:29.592004 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efdafad4-f4a2-4915-8ce6-145762a0a26f None admin] RBAC: Adding query filter params (domain_id=default, name=glance) {{(pid=81578) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:43:29.592380 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efdafad4-f4a2-4915-8ce6-145762a0a26f None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:29.593911 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efdafad4-f4a2-4915-8ce6-145762a0a26f None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:29.612234 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 316/635] 10.222.0.27 () {56 vars in 1137 bytes} [Tue Aug 30 01:43:29 2022] GET /identity/v3/users?domain_id=default&name=glance => generated 372 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:43:29.617518 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4f901bd-a982-4109-b804-e0d5b334eafa None None] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:29.618304 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4f901bd-a982-4109-b804-e0d5b334eafa None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:29.619135 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4f901bd-a982-4109-b804-e0d5b334eafa None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:29.641137 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.auth.core [None req-a4f901bd-a982-4109-b804-e0d5b334eafa None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81578) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 01:43:29.642713 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-a4f901bd-a982-4109-b804-e0d5b334eafa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:43:29.650409 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64d33ab0-5b7f-43d5-add0-819e704181c7 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:29.687536 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2f76c39-c6f6-4751-a0be-b75c7442d4d4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:29.688262 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2f76c39-c6f6-4751-a0be-b75c7442d4d4 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:29.688656 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2f76c39-c6f6-4751-a0be-b75c7442d4d4 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:29.689321 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2f76c39-c6f6-4751-a0be-b75c7442d4d4 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:29.699625 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2f76c39-c6f6-4751-a0be-b75c7442d4d4 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:29.701155 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2f76c39-c6f6-4751-a0be-b75c7442d4d4 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:29.725480 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 317/636] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Aug 30 01:43:29 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 30 01:43:29.725480 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4f901bd-a982-4109-b804-e0d5b334eafa None None] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:29.730449 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application [None req-a2f76c39-c6f6-4751-a0be-b75c7442d4d4 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 01:43:29.730449 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 01:43:29.730449 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 01:43:29.730449 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 01:43:29.730449 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 01:43:29.730449 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 01:43:29.730449 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 01:43:29.730449 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 01:43:29.730449 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 01:43:29.730449 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 01:43:29.730449 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 01:43:29.730449 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 01:43:29.730449 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 30 01:43:29.730449 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 30 01:43:29.730449 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 30 01:43:29.730449 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 30 01:43:29.731304 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 01:43:29.731304 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 01:43:29.731304 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 01:43:29.731304 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 01:43:29.731304 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 01:43:29.731304 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 01:43:29.731304 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 01:43:29.731304 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application with Lock( Aug 30 01:43:29.731304 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 01:43:29.731304 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._enter() Aug 30 01:43:29.731304 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 01:43:29.731304 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 01:43:29.731304 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 01:43:29.731304 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.creator() Aug 30 01:43:29.731304 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 01:43:29.731304 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application created_value = creator( Aug 30 01:43:29.731304 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 30 01:43:29.732284 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 30 01:43:29.732284 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 01:43:29.732284 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Aug 30 01:43:29.734756 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 320/637] 10.222.0.27 () {56 vars in 1108 bytes} [Tue Aug 30 01:43:29 2022] GET /identity/v3/domains/Default => generated 87 bytes in 214 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 01:43:29.735599 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2f76c39-c6f6-4751-a0be-b75c7442d4d4 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:29.794096 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:43:29.812105 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:43:29.847845 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2f76c39-c6f6-4751-a0be-b75c7442d4d4 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:29.857344 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4f901bd-a982-4109-b804-e0d5b334eafa None None] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:29.884995 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a91bdab-c626-4de5-9a2d-8bffeec9d0b5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:29.885743 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a91bdab-c626-4de5-9a2d-8bffeec9d0b5 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:29.886022 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a91bdab-c626-4de5-9a2d-8bffeec9d0b5 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:29.886289 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a91bdab-c626-4de5-9a2d-8bffeec9d0b5 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:29.893240 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37b92978-e54d-48b4-8a92-66d8085ff6c9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:29.893880 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37b92978-e54d-48b4-8a92-66d8085ff6c9 None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:29.894159 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37b92978-e54d-48b4-8a92-66d8085ff6c9 None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:29.894429 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37b92978-e54d-48b4-8a92-66d8085ff6c9 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:29.897535 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a91bdab-c626-4de5-9a2d-8bffeec9d0b5 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 30 01:43:29.903012 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application [None req-0a91bdab-c626-4de5-9a2d-8bffeec9d0b5 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Aug 30 01:43:29.903012 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 01:43:29.903012 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 01:43:29.903012 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 01:43:29.903012 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 01:43:29.903012 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 01:43:29.903012 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 01:43:29.903012 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 01:43:29.903012 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 01:43:29.903012 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 01:43:29.903012 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 01:43:29.903012 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 01:43:29.903012 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 30 01:43:29.903012 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 30 01:43:29.903012 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 30 01:43:29.903012 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 30 01:43:29.904403 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 30 01:43:29.904403 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 30 01:43:29.904403 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 30 01:43:29.904403 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 30 01:43:29.904403 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 01:43:29.904403 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 01:43:29.904403 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 01:43:29.904403 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 01:43:29.904403 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 01:43:29.904403 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 01:43:29.904403 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 01:43:29.904403 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application with Lock( Aug 30 01:43:29.904403 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 01:43:29.904403 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._enter() Aug 30 01:43:29.904403 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 01:43:29.904403 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 01:43:29.905257 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 01:43:29.905257 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.creator() Aug 30 01:43:29.905257 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 01:43:29.905257 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application created_value = creator( Aug 30 01:43:29.905257 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 30 01:43:29.905257 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 30 01:43:29.905257 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 30 01:43:29.905257 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 30 01:43:29.905257 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 30 01:43:29.905257 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 30 01:43:29.905257 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Aug 30 01:43:29.905257 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Aug 30 01:43:29.905257 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 321/638] 10.222.0.27 () {56 vars in 1102 bytes} [Tue Aug 30 01:43:29 2022] GET /identity/v3/roles/service => generated 85 bytes in 169 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 01:43:29.905257 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a91bdab-c626-4de5-9a2d-8bffeec9d0b5 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:29.965844 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37b92978-e54d-48b4-8a92-66d8085ff6c9 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:29.967289 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37b92978-e54d-48b4-8a92-66d8085ff6c9 None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:29.969162 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:43:30.015457 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a91bdab-c626-4de5-9a2d-8bffeec9d0b5 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:30.045590 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:43:30.052800 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a11bb7b0-e79a-4276-97f8-bbafe4782dff None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:30.054249 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a11bb7b0-e79a-4276-97f8-bbafe4782dff None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:30.061040 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a11bb7b0-e79a-4276-97f8-bbafe4782dff None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:30.061040 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a11bb7b0-e79a-4276-97f8-bbafe4782dff None admin] PATH_INFO: `/v3/domains` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:30.061040 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a11bb7b0-e79a-4276-97f8-bbafe4782dff None admin] RBAC: Adding query filter params (name=Default) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:43:30.061040 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a11bb7b0-e79a-4276-97f8-bbafe4782dff None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:30.061040 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a11bb7b0-e79a-4276-97f8-bbafe4782dff None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:30.070395 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 318/639] 10.222.0.27 () {58 vars in 1282 bytes} [Tue Aug 30 01:43:29 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 350 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 30 01:43:30.071241 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37b92978-e54d-48b4-8a92-66d8085ff6c9 None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:30.075608 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 322/640] 10.222.0.27 () {56 vars in 1109 bytes} [Tue Aug 30 01:43:29 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:43:30.076639 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a11bb7b0-e79a-4276-97f8-bbafe4782dff None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:30.180066 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37b92978-e54d-48b4-8a92-66d8085ff6c9 None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:30.181881 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:43:30.189049 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a11bb7b0-e79a-4276-97f8-bbafe4782dff None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:30.216287 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-345cdca3-a316-4d47-8e77-33719b19d046 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:30.216897 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-345cdca3-a316-4d47-8e77-33719b19d046 None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:30.217171 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-345cdca3-a316-4d47-8e77-33719b19d046 None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:30.217439 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-345cdca3-a316-4d47-8e77-33719b19d046 None admin] PATH_INFO: `/v3/roles` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:30.218136 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-345cdca3-a316-4d47-8e77-33719b19d046 None admin] RBAC: Adding query filter params (name=service) {{(pid=81578) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:43:30.218429 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-345cdca3-a316-4d47-8e77-33719b19d046 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:30.219768 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-345cdca3-a316-4d47-8e77-33719b19d046 None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:30.227147 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d76739e3-18f2-493e-b5e7-2c2ad883ec7b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:30.228053 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d76739e3-18f2-493e-b5e7-2c2ad883ec7b None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:30.228347 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 319/641] 10.222.0.27 () {56 vars in 1103 bytes} [Tue Aug 30 01:43:30 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:43:30.228694 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d76739e3-18f2-493e-b5e7-2c2ad883ec7b None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:30.229133 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d76739e3-18f2-493e-b5e7-2c2ad883ec7b None admin] PATH_INFO: `/v3/roles/service` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:30.229246 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-345cdca3-a316-4d47-8e77-33719b19d046 None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:30.244268 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d76739e3-18f2-493e-b5e7-2c2ad883ec7b None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 30 01:43:30.249044 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application [None req-d76739e3-18f2-493e-b5e7-2c2ad883ec7b None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Aug 30 01:43:30.249044 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 01:43:30.249044 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 01:43:30.249044 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 01:43:30.249044 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 01:43:30.249044 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 01:43:30.249044 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 01:43:30.249044 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 01:43:30.249044 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 01:43:30.249044 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 01:43:30.249044 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 01:43:30.249044 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 01:43:30.249044 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 30 01:43:30.249044 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 30 01:43:30.249044 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 30 01:43:30.249044 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 30 01:43:30.250023 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 30 01:43:30.250023 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 30 01:43:30.250023 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 30 01:43:30.250023 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 30 01:43:30.250023 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 01:43:30.250023 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 01:43:30.250023 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 01:43:30.250023 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 01:43:30.250023 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 01:43:30.250023 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 01:43:30.250023 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 01:43:30.250023 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application with Lock( Aug 30 01:43:30.250023 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 01:43:30.250023 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._enter() Aug 30 01:43:30.250023 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 01:43:30.250023 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 01:43:30.250875 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 01:43:30.250875 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.creator() Aug 30 01:43:30.250875 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 01:43:30.250875 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application created_value = creator( Aug 30 01:43:30.250875 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 30 01:43:30.250875 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 30 01:43:30.250875 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 30 01:43:30.250875 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 30 01:43:30.250875 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 30 01:43:30.250875 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 30 01:43:30.250875 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Aug 30 01:43:30.250875 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Aug 30 01:43:30.254985 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 323/642] 10.222.0.27 () {56 vars in 1102 bytes} [Tue Aug 30 01:43:30 2022] GET /identity/v3/roles/service => generated 85 bytes in 179 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 01:43:30.255814 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d76739e3-18f2-493e-b5e7-2c2ad883ec7b None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:30.310972 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:43:30.337381 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-345cdca3-a316-4d47-8e77-33719b19d046 None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:30.362963 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d76739e3-18f2-493e-b5e7-2c2ad883ec7b None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:30.373821 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b0907b2-2aab-497d-96ea-723a2dc52c4b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:30.374464 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b0907b2-2aab-497d-96ea-723a2dc52c4b None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:30.374740 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b0907b2-2aab-497d-96ea-723a2dc52c4b None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:30.375040 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b0907b2-2aab-497d-96ea-723a2dc52c4b None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:30.386481 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b0907b2-2aab-497d-96ea-723a2dc52c4b None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:30.387920 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b0907b2-2aab-497d-96ea-723a2dc52c4b None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:30.402780 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application [None req-9b0907b2-2aab-497d-96ea-723a2dc52c4b None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 01:43:30.402780 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 01:43:30.402780 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 01:43:30.402780 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 01:43:30.402780 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 01:43:30.402780 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 01:43:30.402780 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 01:43:30.402780 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 01:43:30.402780 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 01:43:30.402780 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 01:43:30.402780 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 01:43:30.402780 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 01:43:30.402780 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 30 01:43:30.402780 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 30 01:43:30.402780 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 30 01:43:30.402780 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 30 01:43:30.403819 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 01:43:30.403819 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 01:43:30.403819 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 01:43:30.403819 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 01:43:30.403819 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 01:43:30.403819 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 01:43:30.403819 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 01:43:30.403819 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application with Lock( Aug 30 01:43:30.403819 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 01:43:30.403819 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._enter() Aug 30 01:43:30.403819 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 01:43:30.403819 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 01:43:30.403819 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 01:43:30.403819 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.creator() Aug 30 01:43:30.403819 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 01:43:30.403819 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application created_value = creator( Aug 30 01:43:30.403819 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 30 01:43:30.404757 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 30 01:43:30.404757 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 01:43:30.404757 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application Aug 30 01:43:30.404757 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 320/643] 10.222.0.27 () {56 vars in 1108 bytes} [Tue Aug 30 01:43:30 2022] GET /identity/v3/domains/Default => generated 87 bytes in 175 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 01:43:30.404757 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b0907b2-2aab-497d-96ea-723a2dc52c4b None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:30.405416 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7b29a36-8bbe-4bc0-9809-9d03fbc2cc19 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:30.406093 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7b29a36-8bbe-4bc0-9809-9d03fbc2cc19 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:30.406377 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7b29a36-8bbe-4bc0-9809-9d03fbc2cc19 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:30.406654 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7b29a36-8bbe-4bc0-9809-9d03fbc2cc19 None admin] PATH_INFO: `/v3/users/nova` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:30.416133 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7b29a36-8bbe-4bc0-9809-9d03fbc2cc19 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:30.417652 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7b29a36-8bbe-4bc0-9809-9d03fbc2cc19 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:30.427247 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:43:30.430593 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application [None req-b7b29a36-8bbe-4bc0-9809-9d03fbc2cc19 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Aug 30 01:43:30.430593 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 01:43:30.430593 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 01:43:30.430593 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 01:43:30.430593 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 01:43:30.430593 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 01:43:30.430593 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 01:43:30.430593 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 01:43:30.430593 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 01:43:30.430593 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 01:43:30.430593 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 01:43:30.430593 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 01:43:30.430593 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 30 01:43:30.430593 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 30 01:43:30.430593 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 30 01:43:30.430593 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 30 01:43:30.431570 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 01:43:30.431570 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 01:43:30.431570 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 30 01:43:30.431570 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 30 01:43:30.431570 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 30 01:43:30.431570 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 30 01:43:30.431570 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 01:43:30.431570 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 01:43:30.431570 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 01:43:30.431570 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 01:43:30.431570 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 01:43:30.431570 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application with Lock( Aug 30 01:43:30.431570 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 01:43:30.431570 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._enter() Aug 30 01:43:30.431570 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 01:43:30.431570 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 01:43:30.431570 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 01:43:30.431570 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.creator() Aug 30 01:43:30.432657 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 01:43:30.432657 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application created_value = creator( Aug 30 01:43:30.432657 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 30 01:43:30.432657 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 30 01:43:30.432657 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 30 01:43:30.432657 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 30 01:43:30.432657 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 30 01:43:30.432657 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 30 01:43:30.432657 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Aug 30 01:43:30.432657 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Aug 30 01:43:30.436392 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 324/644] 10.222.0.27 () {56 vars in 1093 bytes} [Tue Aug 30 01:43:30 2022] GET /identity/v3/users/nova => generated 82 bytes in 181 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 01:43:30.437379 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7b29a36-8bbe-4bc0-9809-9d03fbc2cc19 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:30.517326 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b0907b2-2aab-497d-96ea-723a2dc52c4b None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:30.552014 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d510d80-2887-4d70-82be-bdc173687de6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:30.552643 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d510d80-2887-4d70-82be-bdc173687de6 None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:30.553609 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d510d80-2887-4d70-82be-bdc173687de6 None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:30.553954 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d510d80-2887-4d70-82be-bdc173687de6 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:30.558689 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7b29a36-8bbe-4bc0-9809-9d03fbc2cc19 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:30.565392 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d510d80-2887-4d70-82be-bdc173687de6 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:30.566817 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d510d80-2887-4d70-82be-bdc173687de6 None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:30.570538 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:43:30.580882 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application [None req-7d510d80-2887-4d70-82be-bdc173687de6 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 01:43:30.580882 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 01:43:30.580882 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 01:43:30.580882 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 01:43:30.580882 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 01:43:30.580882 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 01:43:30.580882 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 01:43:30.580882 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 01:43:30.580882 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 01:43:30.580882 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 01:43:30.580882 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 01:43:30.580882 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 01:43:30.580882 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 30 01:43:30.580882 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 30 01:43:30.580882 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 30 01:43:30.580882 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 30 01:43:30.582346 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 01:43:30.582346 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 01:43:30.582346 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 01:43:30.582346 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 01:43:30.582346 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 01:43:30.582346 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 01:43:30.582346 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 01:43:30.582346 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application with Lock( Aug 30 01:43:30.582346 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 01:43:30.582346 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._enter() Aug 30 01:43:30.582346 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 01:43:30.582346 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 01:43:30.582346 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 01:43:30.582346 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.creator() Aug 30 01:43:30.582346 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 01:43:30.582346 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application created_value = creator( Aug 30 01:43:30.582346 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 30 01:43:30.583252 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 30 01:43:30.583252 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 01:43:30.583252 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application Aug 30 01:43:30.583252 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 321/645] 10.222.0.27 () {56 vars in 1108 bytes} [Tue Aug 30 01:43:30 2022] GET /identity/v3/domains/Default => generated 87 bytes in 178 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 01:43:30.583252 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d510d80-2887-4d70-82be-bdc173687de6 None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:30.597618 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e5a53a7-0503-4c99-9ec5-1558f35a57d5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:30.598228 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e5a53a7-0503-4c99-9ec5-1558f35a57d5 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:30.598513 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e5a53a7-0503-4c99-9ec5-1558f35a57d5 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:30.598991 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e5a53a7-0503-4c99-9ec5-1558f35a57d5 None admin] PATH_INFO: `/v3/roles` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:30.599452 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e5a53a7-0503-4c99-9ec5-1558f35a57d5 None admin] RBAC: Adding query filter params (name=service) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:43:30.600290 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e5a53a7-0503-4c99-9ec5-1558f35a57d5 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:30.601944 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e5a53a7-0503-4c99-9ec5-1558f35a57d5 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:30.614743 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 325/646] 10.222.0.27 () {56 vars in 1103 bytes} [Tue Aug 30 01:43:30 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:43:30.614743 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e5a53a7-0503-4c99-9ec5-1558f35a57d5 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:30.663279 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:43:30.697536 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d510d80-2887-4d70-82be-bdc173687de6 None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:30.730264 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e5a53a7-0503-4c99-9ec5-1558f35a57d5 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:30.739816 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a73093b7-b90f-4fc7-b29e-3d2ccf0be76e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:30.741422 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a73093b7-b90f-4fc7-b29e-3d2ccf0be76e None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:30.741755 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a73093b7-b90f-4fc7-b29e-3d2ccf0be76e None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:30.742034 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a73093b7-b90f-4fc7-b29e-3d2ccf0be76e None admin] PATH_INFO: `/v3/domains` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:30.742680 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a73093b7-b90f-4fc7-b29e-3d2ccf0be76e None admin] RBAC: Adding query filter params (name=Default) {{(pid=81578) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:43:30.743275 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a73093b7-b90f-4fc7-b29e-3d2ccf0be76e None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:30.744667 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a73093b7-b90f-4fc7-b29e-3d2ccf0be76e None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:30.757187 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 322/647] 10.222.0.27 () {56 vars in 1109 bytes} [Tue Aug 30 01:43:30 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:43:30.762163 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a73093b7-b90f-4fc7-b29e-3d2ccf0be76e None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:30.768701 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a8132d5-a106-4ae4-9367-ff6bea05e30a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:30.769415 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a8132d5-a106-4ae4-9367-ff6bea05e30a None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:30.769839 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a8132d5-a106-4ae4-9367-ff6bea05e30a None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:30.770210 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a8132d5-a106-4ae4-9367-ff6bea05e30a None admin] PATH_INFO: `/v3/users` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:30.771040 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a8132d5-a106-4ae4-9367-ff6bea05e30a None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:43:30.771444 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a8132d5-a106-4ae4-9367-ff6bea05e30a None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:30.773035 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a8132d5-a106-4ae4-9367-ff6bea05e30a None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:30.792094 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 326/648] 10.222.0.27 () {56 vars in 1133 bytes} [Tue Aug 30 01:43:30 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:43:30.797272 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a8132d5-a106-4ae4-9367-ff6bea05e30a None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:30.831107 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:43:30.875742 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a73093b7-b90f-4fc7-b29e-3d2ccf0be76e None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:30.905660 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:43:30.913845 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a8132d5-a106-4ae4-9367-ff6bea05e30a None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:30.914466 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcf3b002-8aef-4039-8660-306ff668ab4b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:30.915073 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcf3b002-8aef-4039-8660-306ff668ab4b None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:30.918334 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcf3b002-8aef-4039-8660-306ff668ab4b None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:30.918334 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcf3b002-8aef-4039-8660-306ff668ab4b None admin] PATH_INFO: `/v3/domains` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:30.918334 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fcf3b002-8aef-4039-8660-306ff668ab4b None admin] RBAC: Adding query filter params (name=Default) {{(pid=81578) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:43:30.918334 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fcf3b002-8aef-4039-8660-306ff668ab4b None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:30.918334 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fcf3b002-8aef-4039-8660-306ff668ab4b None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:30.930225 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 323/649] 10.222.0.27 () {56 vars in 1109 bytes} [Tue Aug 30 01:43:30 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:43:30.931290 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcf3b002-8aef-4039-8660-306ff668ab4b None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:30.956903 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-669b4653-9a3b-4980-8115-7b94ab07494c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:30.957572 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-669b4653-9a3b-4980-8115-7b94ab07494c None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:30.957854 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-669b4653-9a3b-4980-8115-7b94ab07494c None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:30.958125 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-669b4653-9a3b-4980-8115-7b94ab07494c None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:30.969126 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-669b4653-9a3b-4980-8115-7b94ab07494c None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:30.970609 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-669b4653-9a3b-4980-8115-7b94ab07494c None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:30.987653 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application [None req-669b4653-9a3b-4980-8115-7b94ab07494c None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 01:43:30.987653 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 01:43:30.987653 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 01:43:30.987653 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 01:43:30.987653 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 01:43:30.987653 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 01:43:30.987653 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 01:43:30.987653 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 01:43:30.987653 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 01:43:30.987653 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 01:43:30.987653 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 01:43:30.987653 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 01:43:30.987653 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 30 01:43:30.987653 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 30 01:43:30.987653 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 30 01:43:30.987653 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 30 01:43:30.988813 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 01:43:30.988813 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 01:43:30.988813 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 01:43:30.988813 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 01:43:30.988813 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 01:43:30.988813 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 01:43:30.988813 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 01:43:30.988813 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application with Lock( Aug 30 01:43:30.988813 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 01:43:30.988813 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._enter() Aug 30 01:43:30.988813 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 01:43:30.988813 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 01:43:30.988813 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 01:43:30.988813 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.creator() Aug 30 01:43:30.988813 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 01:43:30.988813 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application created_value = creator( Aug 30 01:43:30.988813 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 30 01:43:30.989940 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 30 01:43:30.989940 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 01:43:30.989940 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Aug 30 01:43:30.995943 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 327/650] 10.222.0.27 () {56 vars in 1108 bytes} [Tue Aug 30 01:43:30 2022] GET /identity/v3/domains/Default => generated 87 bytes in 199 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 01:43:30.995943 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-669b4653-9a3b-4980-8115-7b94ab07494c None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:31.048939 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcf3b002-8aef-4039-8660-306ff668ab4b None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:31.084729 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:43:31.087794 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb2b53a7-6bdc-4a5f-8ef8-fafb1d9904a0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:31.088496 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb2b53a7-6bdc-4a5f-8ef8-fafb1d9904a0 None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:31.088775 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb2b53a7-6bdc-4a5f-8ef8-fafb1d9904a0 None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:31.089046 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb2b53a7-6bdc-4a5f-8ef8-fafb1d9904a0 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:31.096146 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb2b53a7-6bdc-4a5f-8ef8-fafb1d9904a0 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:31.097606 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb2b53a7-6bdc-4a5f-8ef8-fafb1d9904a0 None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:31.103169 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-669b4653-9a3b-4980-8115-7b94ab07494c None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:31.109298 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application [None req-eb2b53a7-6bdc-4a5f-8ef8-fafb1d9904a0 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 30 01:43:31.109298 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 01:43:31.109298 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 01:43:31.109298 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 01:43:31.109298 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 01:43:31.109298 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 01:43:31.109298 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 01:43:31.109298 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 01:43:31.109298 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 01:43:31.109298 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 01:43:31.109298 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 01:43:31.109298 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 01:43:31.109298 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 30 01:43:31.109298 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 30 01:43:31.109298 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 30 01:43:31.110218 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 30 01:43:31.110218 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 01:43:31.110218 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 01:43:31.110218 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 01:43:31.110218 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 01:43:31.110218 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 01:43:31.110218 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 01:43:31.110218 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 01:43:31.110218 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application with Lock( Aug 30 01:43:31.110218 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 01:43:31.110218 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._enter() Aug 30 01:43:31.110218 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 01:43:31.110218 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 01:43:31.110218 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 01:43:31.110218 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.creator() Aug 30 01:43:31.110218 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 01:43:31.110218 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application created_value = creator( Aug 30 01:43:31.111101 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 30 01:43:31.111101 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 30 01:43:31.111101 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 30 01:43:31.111101 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 30 01:43:31.111101 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 30 01:43:31.111101 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 30 01:43:31.111101 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 30 01:43:31.111101 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application Aug 30 01:43:31.111101 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 324/651] 10.222.0.27 () {56 vars in 1111 bytes} [Tue Aug 30 01:43:30 2022] GET /identity/v3/projects/service => generated 88 bytes in 179 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 01:43:31.111586 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb2b53a7-6bdc-4a5f-8ef8-fafb1d9904a0 None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:31.144834 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47bc8fb4-1847-46ea-a950-b16dff336212 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:31.145446 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47bc8fb4-1847-46ea-a950-b16dff336212 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:31.145757 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47bc8fb4-1847-46ea-a950-b16dff336212 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:31.146031 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47bc8fb4-1847-46ea-a950-b16dff336212 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:31.162317 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:43:31.220884 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47bc8fb4-1847-46ea-a950-b16dff336212 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:31.221627 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb2b53a7-6bdc-4a5f-8ef8-fafb1d9904a0 None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:31.222362 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47bc8fb4-1847-46ea-a950-b16dff336212 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:31.260454 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-199a5738-9dfb-4c3b-a763-3548a59e1cb5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:31.261090 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-199a5738-9dfb-4c3b-a763-3548a59e1cb5 None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:31.261370 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-199a5738-9dfb-4c3b-a763-3548a59e1cb5 None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:31.261668 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-199a5738-9dfb-4c3b-a763-3548a59e1cb5 None admin] PATH_INFO: `/v3/users/cinder` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:31.269237 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-199a5738-9dfb-4c3b-a763-3548a59e1cb5 None admin] RBAC: Authorizing `identity:get_user(user_id=cinder)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:31.270702 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-199a5738-9dfb-4c3b-a763-3548a59e1cb5 None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:31.282925 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application [None req-199a5738-9dfb-4c3b-a763-3548a59e1cb5 None admin] Could not find user: cinder.: keystone.exception.UserNotFound: Could not find user: cinder. Aug 30 01:43:31.282925 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 01:43:31.282925 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 01:43:31.282925 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 01:43:31.282925 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 01:43:31.282925 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 01:43:31.282925 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 01:43:31.282925 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 01:43:31.282925 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 01:43:31.282925 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 01:43:31.282925 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 01:43:31.282925 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 01:43:31.282925 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 30 01:43:31.282925 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 30 01:43:31.282925 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 30 01:43:31.282925 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 30 01:43:31.283985 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 01:43:31.283985 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 01:43:31.283985 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 30 01:43:31.283985 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 30 01:43:31.283985 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 30 01:43:31.283985 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 30 01:43:31.283985 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 01:43:31.283985 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 01:43:31.283985 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 01:43:31.283985 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 01:43:31.283985 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 01:43:31.283985 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application with Lock( Aug 30 01:43:31.283985 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 01:43:31.283985 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._enter() Aug 30 01:43:31.283985 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 01:43:31.283985 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 01:43:31.283985 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 01:43:31.283985 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.creator() Aug 30 01:43:31.284924 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 01:43:31.284924 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application created_value = creator( Aug 30 01:43:31.284924 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 30 01:43:31.284924 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 30 01:43:31.284924 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 30 01:43:31.284924 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 30 01:43:31.284924 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 30 01:43:31.284924 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 30 01:43:31.284924 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: cinder. Aug 30 01:43:31.284924 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application Aug 30 01:43:31.284924 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 325/652] 10.222.0.27 () {56 vars in 1099 bytes} [Tue Aug 30 01:43:31 2022] GET /identity/v3/users/cinder => generated 84 bytes in 173 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 01:43:31.285610 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-199a5738-9dfb-4c3b-a763-3548a59e1cb5 None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:31.316362 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:43:31.335704 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 328/653] 10.222.0.27 () {58 vars in 1282 bytes} [Tue Aug 30 01:43:30 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 344 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 30 01:43:31.336523 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47bc8fb4-1847-46ea-a950-b16dff336212 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:31.387270 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-199a5738-9dfb-4c3b-a763-3548a59e1cb5 None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:31.395860 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:43:31.420511 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d1f6240-46a3-4130-b2f6-72074fed5e2c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:31.421114 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d1f6240-46a3-4130-b2f6-72074fed5e2c None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:31.421392 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d1f6240-46a3-4130-b2f6-72074fed5e2c None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:31.421693 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d1f6240-46a3-4130-b2f6-72074fed5e2c None admin] PATH_INFO: `/v3/domains` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:31.422345 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d1f6240-46a3-4130-b2f6-72074fed5e2c None admin] RBAC: Adding query filter params (name=Default) {{(pid=81578) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:43:31.423064 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d1f6240-46a3-4130-b2f6-72074fed5e2c None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:31.424978 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d1f6240-46a3-4130-b2f6-72074fed5e2c None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:31.435678 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47bc8fb4-1847-46ea-a950-b16dff336212 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:31.437374 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 326/654] 10.222.0.27 () {56 vars in 1109 bytes} [Tue Aug 30 01:43:31 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:43:31.438192 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d1f6240-46a3-4130-b2f6-72074fed5e2c None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:31.471427 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8209d283-8a1e-45b1-b263-37da24702126 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:31.472032 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8209d283-8a1e-45b1-b263-37da24702126 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:31.472311 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8209d283-8a1e-45b1-b263-37da24702126 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:31.472582 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8209d283-8a1e-45b1-b263-37da24702126 None admin] PATH_INFO: `/v3/projects` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:31.473272 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8209d283-8a1e-45b1-b263-37da24702126 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:43:31.473616 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8209d283-8a1e-45b1-b263-37da24702126 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:31.475049 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8209d283-8a1e-45b1-b263-37da24702126 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:31.488062 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 329/655] 10.222.0.27 () {56 vars in 1148 bytes} [Tue Aug 30 01:43:31 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:43:31.489230 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-567ba8d7-007d-4488-a5e3-4a1743d83198 None None] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:31.489554 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-567ba8d7-007d-4488-a5e3-4a1743d83198 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:31.489834 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-567ba8d7-007d-4488-a5e3-4a1743d83198 None None] PATH_INFO: `/` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:31.491330 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 330/656] 10.222.0.27 () {54 vars in 927 bytes} [Tue Aug 30 01:43:31 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 01:43:31.492151 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-567ba8d7-007d-4488-a5e3-4a1743d83198 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:31.541217 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:43:31.549223 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d1f6240-46a3-4130-b2f6-72074fed5e2c None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:31.584967 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-880a5ff0-4675-4bd9-b78c-76c3cb614807 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:31.585879 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-880a5ff0-4675-4bd9-b78c-76c3cb614807 None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:31.585879 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-880a5ff0-4675-4bd9-b78c-76c3cb614807 None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:31.586123 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-880a5ff0-4675-4bd9-b78c-76c3cb614807 None admin] PATH_INFO: `/v3/users` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:31.586850 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-880a5ff0-4675-4bd9-b78c-76c3cb614807 None admin] RBAC: Adding query filter params (domain_id=default, name=cinder) {{(pid=81578) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:43:31.587156 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-880a5ff0-4675-4bd9-b78c-76c3cb614807 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:31.588564 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-880a5ff0-4675-4bd9-b78c-76c3cb614807 None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:31.598490 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-567ba8d7-007d-4488-a5e3-4a1743d83198 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:31.605789 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 327/657] 10.222.0.27 () {56 vars in 1137 bytes} [Tue Aug 30 01:43:31 2022] GET /identity/v3/users?domain_id=default&name=cinder => generated 372 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:43:31.606587 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-880a5ff0-4675-4bd9-b78c-76c3cb614807 None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:31.634046 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:43:31.636750 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-158555b5-fb66-446e-a8be-6f946cfdb358 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:31.637379 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-158555b5-fb66-446e-a8be-6f946cfdb358 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:31.637690 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-158555b5-fb66-446e-a8be-6f946cfdb358 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:31.637967 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-158555b5-fb66-446e-a8be-6f946cfdb358 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:31.648975 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-158555b5-fb66-446e-a8be-6f946cfdb358 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:31.650479 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-158555b5-fb66-446e-a8be-6f946cfdb358 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:31.664159 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application [None req-158555b5-fb66-446e-a8be-6f946cfdb358 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 01:43:31.664159 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 01:43:31.664159 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 01:43:31.664159 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 01:43:31.664159 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 01:43:31.664159 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 01:43:31.664159 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 01:43:31.664159 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 01:43:31.664159 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 01:43:31.664159 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 01:43:31.664159 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 01:43:31.664159 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 01:43:31.664159 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 30 01:43:31.664159 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 30 01:43:31.664159 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 30 01:43:31.664159 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 30 01:43:31.665254 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 01:43:31.665254 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 01:43:31.665254 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 01:43:31.665254 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 01:43:31.665254 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 01:43:31.665254 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 01:43:31.665254 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 01:43:31.665254 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application with Lock( Aug 30 01:43:31.665254 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 01:43:31.665254 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._enter() Aug 30 01:43:31.665254 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 01:43:31.665254 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 01:43:31.665254 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 01:43:31.665254 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.creator() Aug 30 01:43:31.665254 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 01:43:31.665254 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application created_value = creator( Aug 30 01:43:31.665254 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 30 01:43:31.666355 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 30 01:43:31.666355 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 01:43:31.666355 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Aug 30 01:43:31.666355 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 331/658] 10.222.0.27 () {56 vars in 1108 bytes} [Tue Aug 30 01:43:31 2022] GET /identity/v3/domains/Default => generated 87 bytes in 174 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 01:43:31.666355 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-158555b5-fb66-446e-a8be-6f946cfdb358 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:31.710793 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-880a5ff0-4675-4bd9-b78c-76c3cb614807 None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:31.746100 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-727540ef-2951-4de8-a1c1-82fcc0a4887c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:31.746754 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-727540ef-2951-4de8-a1c1-82fcc0a4887c None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:31.747028 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-727540ef-2951-4de8-a1c1-82fcc0a4887c None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:31.747293 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-727540ef-2951-4de8-a1c1-82fcc0a4887c None admin] PATH_INFO: `/v3/users/neutron` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:31.754305 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-727540ef-2951-4de8-a1c1-82fcc0a4887c None admin] RBAC: Authorizing `identity:get_user(user_id=neutron)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:31.755729 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-727540ef-2951-4de8-a1c1-82fcc0a4887c None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:31.762104 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:43:31.767759 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application [None req-727540ef-2951-4de8-a1c1-82fcc0a4887c None admin] Could not find user: neutron.: keystone.exception.UserNotFound: Could not find user: neutron. Aug 30 01:43:31.767759 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 01:43:31.767759 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 01:43:31.767759 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 01:43:31.767759 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 01:43:31.767759 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 01:43:31.767759 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 01:43:31.767759 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 01:43:31.767759 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 01:43:31.767759 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 01:43:31.767759 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 01:43:31.767759 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 01:43:31.767759 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 30 01:43:31.767759 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 30 01:43:31.767759 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 30 01:43:31.767759 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 30 01:43:31.775843 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 01:43:31.775843 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 01:43:31.775843 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 30 01:43:31.775843 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 30 01:43:31.775843 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 30 01:43:31.775843 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 30 01:43:31.775843 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 01:43:31.775843 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 01:43:31.775843 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 01:43:31.775843 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 01:43:31.775843 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 01:43:31.775843 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application with Lock( Aug 30 01:43:31.775843 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 01:43:31.775843 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._enter() Aug 30 01:43:31.775843 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 01:43:31.775843 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 01:43:31.775843 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 01:43:31.775843 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.creator() Aug 30 01:43:31.776885 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 01:43:31.776885 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application created_value = creator( Aug 30 01:43:31.776885 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 30 01:43:31.776885 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 30 01:43:31.776885 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 30 01:43:31.776885 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 30 01:43:31.776885 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 30 01:43:31.776885 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 30 01:43:31.776885 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: neutron. Aug 30 01:43:31.776885 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application Aug 30 01:43:31.776885 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 328/659] 10.222.0.27 () {56 vars in 1102 bytes} [Tue Aug 30 01:43:31 2022] GET /identity/v3/users/neutron => generated 85 bytes in 162 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 01:43:31.777645 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-158555b5-fb66-446e-a8be-6f946cfdb358 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:31.777730 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a33982c4-aa88-4c06-8707-997baa379817 None None] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:31.777730 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a33982c4-aa88-4c06-8707-997baa379817 None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:31.777730 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a33982c4-aa88-4c06-8707-997baa379817 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:31.789563 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.auth.core [None req-a33982c4-aa88-4c06-8707-997baa379817 None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81578) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 01:43:31.790765 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-a33982c4-aa88-4c06-8707-997baa379817 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:43:31.804893 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06609075-e37d-4348-aeee-fe87675de11e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:31.805557 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06609075-e37d-4348-aeee-fe87675de11e None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:31.805836 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06609075-e37d-4348-aeee-fe87675de11e None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:31.806103 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06609075-e37d-4348-aeee-fe87675de11e None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:31.806821 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06609075-e37d-4348-aeee-fe87675de11e None admin] RBAC: Adding query filter params (role.id=aa59b0c9009a480994d9df81a5f8d404, scope.project.id=4562f3951fbe465280073e60a11e44e2, user.id=3caff62093d748f48596574836e12e42) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:43:31.807123 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06609075-e37d-4348-aeee-fe87675de11e None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:31.808561 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06609075-e37d-4348-aeee-fe87675de11e None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:31.814190 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 332/660] 10.222.0.27 () {56 vars in 1374 bytes} [Tue Aug 30 01:43:31 2022] GET /identity/v3/role_assignments?user.id=3caff62093d748f48596574836e12e42&scope.project.id=4562f3951fbe465280073e60a11e44e2&role.id=aa59b0c9009a480994d9df81a5f8d404 => generated 608 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:43:31.814949 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06609075-e37d-4348-aeee-fe87675de11e None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:31.868858 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 329/661] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Aug 30 01:43:31 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 30 01:43:31.869788 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a33982c4-aa88-4c06-8707-997baa379817 None None] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:31.906847 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:43:31.944189 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06609075-e37d-4348-aeee-fe87675de11e None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:31.984363 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91b55cd5-599b-4f85-a71c-b92f17860fbc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:31.985215 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91b55cd5-599b-4f85-a71c-b92f17860fbc None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:31.985787 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91b55cd5-599b-4f85-a71c-b92f17860fbc None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:31.986192 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91b55cd5-599b-4f85-a71c-b92f17860fbc None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:32.013873 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a33982c4-aa88-4c06-8707-997baa379817 None None] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:32.025996 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:43:32.060502 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8982d40-0708-477b-8231-25653a60687c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:32.061112 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8982d40-0708-477b-8231-25653a60687c None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:32.062089 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8982d40-0708-477b-8231-25653a60687c None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:32.062089 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8982d40-0708-477b-8231-25653a60687c None admin] PATH_INFO: `/v3/domains` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:32.062466 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8982d40-0708-477b-8231-25653a60687c None admin] RBAC: Adding query filter params (name=Default) {{(pid=81578) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:43:32.063476 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8982d40-0708-477b-8231-25653a60687c None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:32.064966 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8982d40-0708-477b-8231-25653a60687c None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:32.077215 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91b55cd5-599b-4f85-a71c-b92f17860fbc None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:32.078588 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 330/662] 10.222.0.27 () {56 vars in 1109 bytes} [Tue Aug 30 01:43:31 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 209 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:43:32.078718 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91b55cd5-599b-4f85-a71c-b92f17860fbc None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:32.079651 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8982d40-0708-477b-8231-25653a60687c None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:32.174043 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:43:32.199757 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8982d40-0708-477b-8231-25653a60687c None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:32.204358 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 333/663] 10.222.0.27 () {58 vars in 1282 bytes} [Tue Aug 30 01:43:31 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 385 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 30 01:43:32.204358 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91b55cd5-599b-4f85-a71c-b92f17860fbc None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:32.240325 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf6bdcc7-4831-45ad-b60b-4ec062b25fc0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:32.241607 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf6bdcc7-4831-45ad-b60b-4ec062b25fc0 None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:32.241884 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf6bdcc7-4831-45ad-b60b-4ec062b25fc0 None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:32.242151 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf6bdcc7-4831-45ad-b60b-4ec062b25fc0 None admin] PATH_INFO: `/v3/users` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:32.242879 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf6bdcc7-4831-45ad-b60b-4ec062b25fc0 None admin] RBAC: Adding query filter params (domain_id=default, name=neutron) {{(pid=81578) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:43:32.243181 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf6bdcc7-4831-45ad-b60b-4ec062b25fc0 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:32.244698 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf6bdcc7-4831-45ad-b60b-4ec062b25fc0 None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:32.266491 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:43:32.267352 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 331/664] 10.222.0.27 () {56 vars in 1139 bytes} [Tue Aug 30 01:43:32 2022] GET /identity/v3/users?domain_id=default&name=neutron => generated 374 bytes in 188 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:43:32.268139 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf6bdcc7-4831-45ad-b60b-4ec062b25fc0 None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:32.308711 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91b55cd5-599b-4f85-a71c-b92f17860fbc None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:32.343003 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2daf0a4c-4f95-40f4-b0fc-5dddc9935efc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:32.344244 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2daf0a4c-4f95-40f4-b0fc-5dddc9935efc None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:32.344244 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2daf0a4c-4f95-40f4-b0fc-5dddc9935efc None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:32.344244 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2daf0a4c-4f95-40f4-b0fc-5dddc9935efc None admin] PATH_INFO: `/v3/projects/service` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:32.351901 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2daf0a4c-4f95-40f4-b0fc-5dddc9935efc None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:32.353432 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2daf0a4c-4f95-40f4-b0fc-5dddc9935efc None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:32.365128 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application [None req-2daf0a4c-4f95-40f4-b0fc-5dddc9935efc None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 30 01:43:32.365128 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 01:43:32.365128 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 01:43:32.365128 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 01:43:32.365128 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 01:43:32.365128 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 01:43:32.365128 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 01:43:32.365128 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 01:43:32.365128 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 01:43:32.365128 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 01:43:32.365128 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 01:43:32.365128 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 01:43:32.365128 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 30 01:43:32.365128 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 30 01:43:32.365128 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 30 01:43:32.367041 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 30 01:43:32.367041 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 01:43:32.367041 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 01:43:32.367041 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 01:43:32.367041 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 01:43:32.367041 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 01:43:32.367041 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 01:43:32.367041 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 01:43:32.367041 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application with Lock( Aug 30 01:43:32.367041 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 01:43:32.367041 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._enter() Aug 30 01:43:32.367041 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 01:43:32.367041 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 01:43:32.367041 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 01:43:32.367041 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.creator() Aug 30 01:43:32.367041 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 01:43:32.367041 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application created_value = creator( Aug 30 01:43:32.368441 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 30 01:43:32.368441 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 30 01:43:32.368441 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 30 01:43:32.368441 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 30 01:43:32.368441 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 30 01:43:32.368441 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 30 01:43:32.368441 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 30 01:43:32.368441 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Aug 30 01:43:32.368441 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 334/665] 10.222.0.27 () {56 vars in 1111 bytes} [Tue Aug 30 01:43:32 2022] GET /identity/v3/projects/service => generated 88 bytes in 166 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 01:43:32.368441 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2daf0a4c-4f95-40f4-b0fc-5dddc9935efc None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:32.382644 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf6bdcc7-4831-45ad-b60b-4ec062b25fc0 None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:32.418705 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4fdd214-4273-4c93-b215-bacc585e2b2c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:32.419341 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4fdd214-4273-4c93-b215-bacc585e2b2c None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:32.419621 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4fdd214-4273-4c93-b215-bacc585e2b2c None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:32.419883 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4fdd214-4273-4c93-b215-bacc585e2b2c None admin] PATH_INFO: `/v3/roles/service` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:32.421882 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:43:32.433261 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4fdd214-4273-4c93-b215-bacc585e2b2c None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 30 01:43:32.436529 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application [None req-b4fdd214-4273-4c93-b215-bacc585e2b2c None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Aug 30 01:43:32.436529 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 01:43:32.436529 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 01:43:32.436529 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 01:43:32.436529 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 01:43:32.436529 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 01:43:32.436529 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 01:43:32.436529 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 01:43:32.436529 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 01:43:32.436529 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 01:43:32.436529 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 01:43:32.436529 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 01:43:32.436529 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 30 01:43:32.436529 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 30 01:43:32.436529 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 30 01:43:32.436529 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 30 01:43:32.437505 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 30 01:43:32.437505 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 30 01:43:32.437505 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 30 01:43:32.437505 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 30 01:43:32.437505 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 01:43:32.437505 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 01:43:32.437505 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 01:43:32.437505 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 01:43:32.437505 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 01:43:32.437505 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 01:43:32.437505 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 01:43:32.437505 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application with Lock( Aug 30 01:43:32.437505 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 01:43:32.437505 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._enter() Aug 30 01:43:32.437505 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 01:43:32.437505 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 01:43:32.438391 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 01:43:32.438391 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.creator() Aug 30 01:43:32.438391 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 01:43:32.438391 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application created_value = creator( Aug 30 01:43:32.438391 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 30 01:43:32.438391 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 30 01:43:32.438391 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 30 01:43:32.438391 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 30 01:43:32.438391 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 30 01:43:32.438391 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 30 01:43:32.438391 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Aug 30 01:43:32.438391 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application Aug 30 01:43:32.438391 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 332/666] 10.222.0.27 () {56 vars in 1102 bytes} [Tue Aug 30 01:43:32 2022] GET /identity/v3/roles/service => generated 85 bytes in 170 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 01:43:32.438391 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4fdd214-4273-4c93-b215-bacc585e2b2c None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:32.483375 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2daf0a4c-4f95-40f4-b0fc-5dddc9935efc None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:32.501299 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:43:32.518589 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0169239-4361-471c-be23-909e9e7b77d7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:32.519220 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0169239-4361-471c-be23-909e9e7b77d7 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:32.519502 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0169239-4361-471c-be23-909e9e7b77d7 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:32.519775 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0169239-4361-471c-be23-909e9e7b77d7 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:32.531169 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0169239-4361-471c-be23-909e9e7b77d7 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:32.533129 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0169239-4361-471c-be23-909e9e7b77d7 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:32.547805 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4fdd214-4273-4c93-b215-bacc585e2b2c None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:32.548958 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application [None req-c0169239-4361-471c-be23-909e9e7b77d7 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 01:43:32.548958 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 01:43:32.548958 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 01:43:32.548958 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 01:43:32.548958 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 01:43:32.548958 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 01:43:32.548958 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 01:43:32.548958 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 01:43:32.548958 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 01:43:32.548958 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 01:43:32.548958 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 01:43:32.548958 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 01:43:32.548958 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 30 01:43:32.548958 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 30 01:43:32.548958 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 30 01:43:32.548958 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 30 01:43:32.549946 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 01:43:32.549946 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 01:43:32.549946 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 01:43:32.549946 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 01:43:32.549946 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 01:43:32.549946 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 01:43:32.549946 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 01:43:32.549946 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application with Lock( Aug 30 01:43:32.549946 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 01:43:32.549946 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._enter() Aug 30 01:43:32.549946 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 01:43:32.549946 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 01:43:32.549946 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 01:43:32.549946 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.creator() Aug 30 01:43:32.549946 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 01:43:32.549946 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application created_value = creator( Aug 30 01:43:32.549946 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 30 01:43:32.550839 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 30 01:43:32.550839 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 01:43:32.550839 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Aug 30 01:43:32.550839 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 335/667] 10.222.0.27 () {56 vars in 1108 bytes} [Tue Aug 30 01:43:32 2022] GET /identity/v3/domains/Default => generated 87 bytes in 183 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 01:43:32.550839 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0169239-4361-471c-be23-909e9e7b77d7 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:32.585276 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-795dd691-3a2c-4174-bee3-ed82be2e0f2f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:32.586076 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-795dd691-3a2c-4174-bee3-ed82be2e0f2f None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:32.586435 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-795dd691-3a2c-4174-bee3-ed82be2e0f2f None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:32.586802 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-795dd691-3a2c-4174-bee3-ed82be2e0f2f None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:32.649686 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:43:32.657768 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-795dd691-3a2c-4174-bee3-ed82be2e0f2f None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:32.659232 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-795dd691-3a2c-4174-bee3-ed82be2e0f2f None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:32.666745 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0169239-4361-471c-be23-909e9e7b77d7 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:32.706081 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-972900cf-daf5-4383-bde1-f1e793d16197 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:32.707313 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-972900cf-daf5-4383-bde1-f1e793d16197 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:32.707596 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-972900cf-daf5-4383-bde1-f1e793d16197 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:32.708064 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-972900cf-daf5-4383-bde1-f1e793d16197 None admin] PATH_INFO: `/v3/projects` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:32.709049 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-972900cf-daf5-4383-bde1-f1e793d16197 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:43:32.709983 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-972900cf-daf5-4383-bde1-f1e793d16197 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:32.711405 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-972900cf-daf5-4383-bde1-f1e793d16197 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:32.724682 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 336/668] 10.222.0.27 () {56 vars in 1148 bytes} [Tue Aug 30 01:43:32 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:43:32.726277 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-972900cf-daf5-4383-bde1-f1e793d16197 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:32.765369 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:43:32.774175 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 333/669] 10.222.0.27 () {58 vars in 1282 bytes} [Tue Aug 30 01:43:32 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 331 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 30 01:43:32.774175 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-795dd691-3a2c-4174-bee3-ed82be2e0f2f None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:32.853945 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-972900cf-daf5-4383-bde1-f1e793d16197 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:33.027993 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:43:33.027993 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-795dd691-3a2c-4174-bee3-ed82be2e0f2f None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:33.028249 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17be369b-7acc-437b-bb16-53f8ad2a246c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:33.028249 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17be369b-7acc-437b-bb16-53f8ad2a246c None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:33.028249 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17be369b-7acc-437b-bb16-53f8ad2a246c None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:33.028249 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17be369b-7acc-437b-bb16-53f8ad2a246c None admin] PATH_INFO: `/v3/roles` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:33.028877 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17be369b-7acc-437b-bb16-53f8ad2a246c None admin] RBAC: Adding query filter params (name=service) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:43:33.028877 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17be369b-7acc-437b-bb16-53f8ad2a246c None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:33.028877 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17be369b-7acc-437b-bb16-53f8ad2a246c None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:33.028877 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 337/670] 10.222.0.27 () {56 vars in 1103 bytes} [Tue Aug 30 01:43:32 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:43:33.028877 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17be369b-7acc-437b-bb16-53f8ad2a246c None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:33.029214 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96d0d5c5-f9ff-498f-9ac1-83b076923972 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:33.029214 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96d0d5c5-f9ff-498f-9ac1-83b076923972 None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:33.029214 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96d0d5c5-f9ff-498f-9ac1-83b076923972 None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:33.029214 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96d0d5c5-f9ff-498f-9ac1-83b076923972 None admin] PATH_INFO: `/v3/domains` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:33.029537 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96d0d5c5-f9ff-498f-9ac1-83b076923972 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81578) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:43:33.029537 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96d0d5c5-f9ff-498f-9ac1-83b076923972 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:33.029537 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96d0d5c5-f9ff-498f-9ac1-83b076923972 None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:33.029537 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 334/671] 10.222.0.27 () {56 vars in 1109 bytes} [Tue Aug 30 01:43:32 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 180 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:43:33.029537 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96d0d5c5-f9ff-498f-9ac1-83b076923972 None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:33.029865 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:43:33.029865 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17be369b-7acc-437b-bb16-53f8ad2a246c None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:33.056717 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96d0d5c5-f9ff-498f-9ac1-83b076923972 None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:33.059866 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-587f27fe-447f-47fd-a516-93d7b1eb52cd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:33.060477 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-587f27fe-447f-47fd-a516-93d7b1eb52cd None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:33.060753 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-587f27fe-447f-47fd-a516-93d7b1eb52cd None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:33.061021 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-587f27fe-447f-47fd-a516-93d7b1eb52cd None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:33.061772 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-587f27fe-447f-47fd-a516-93d7b1eb52cd None admin] RBAC: Adding query filter params (role.id=aa59b0c9009a480994d9df81a5f8d404, scope.project.id=4562f3951fbe465280073e60a11e44e2, user.id=d64f2e71361644589bec8a06e6eceed1) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:43:33.062082 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-587f27fe-447f-47fd-a516-93d7b1eb52cd None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:33.063911 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-587f27fe-447f-47fd-a516-93d7b1eb52cd None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:33.073613 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 338/672] 10.222.0.27 () {56 vars in 1374 bytes} [Tue Aug 30 01:43:32 2022] GET /identity/v3/role_assignments?user.id=d64f2e71361644589bec8a06e6eceed1&scope.project.id=4562f3951fbe465280073e60a11e44e2&role.id=aa59b0c9009a480994d9df81a5f8d404 => generated 608 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:43:33.074456 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-587f27fe-447f-47fd-a516-93d7b1eb52cd None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:33.108694 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-431871fd-de52-4a2f-86a4-36c750400c80 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:33.110359 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-431871fd-de52-4a2f-86a4-36c750400c80 None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:33.111013 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-431871fd-de52-4a2f-86a4-36c750400c80 None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:33.111631 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-431871fd-de52-4a2f-86a4-36c750400c80 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:33.127557 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-431871fd-de52-4a2f-86a4-36c750400c80 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:33.130275 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-431871fd-de52-4a2f-86a4-36c750400c80 None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:33.150407 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application [None req-431871fd-de52-4a2f-86a4-36c750400c80 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 01:43:33.150407 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 01:43:33.150407 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 01:43:33.150407 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 01:43:33.150407 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 01:43:33.150407 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 01:43:33.150407 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 01:43:33.150407 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 01:43:33.150407 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 01:43:33.150407 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 01:43:33.150407 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 01:43:33.150407 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 01:43:33.150407 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 30 01:43:33.150407 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 30 01:43:33.150407 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 30 01:43:33.150407 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 30 01:43:33.151590 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 01:43:33.151590 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 01:43:33.151590 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 01:43:33.151590 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 01:43:33.151590 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 01:43:33.151590 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 01:43:33.151590 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 01:43:33.151590 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application with Lock( Aug 30 01:43:33.151590 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 01:43:33.151590 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._enter() Aug 30 01:43:33.151590 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 01:43:33.151590 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 01:43:33.151590 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 01:43:33.151590 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.creator() Aug 30 01:43:33.151590 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 01:43:33.151590 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application created_value = creator( Aug 30 01:43:33.151590 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 30 01:43:33.152555 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 30 01:43:33.152555 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 01:43:33.152555 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application Aug 30 01:43:33.155124 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 335/673] 10.222.0.27 () {56 vars in 1108 bytes} [Tue Aug 30 01:43:32 2022] GET /identity/v3/domains/Default => generated 87 bytes in 204 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 01:43:33.156554 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-431871fd-de52-4a2f-86a4-36c750400c80 None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:33.198834 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-587f27fe-447f-47fd-a516-93d7b1eb52cd None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:33.238843 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1753d26f-9ad2-4041-92ab-608b35328d0c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:33.239505 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1753d26f-9ad2-4041-92ab-608b35328d0c None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:33.239780 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1753d26f-9ad2-4041-92ab-608b35328d0c None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:33.240046 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1753d26f-9ad2-4041-92ab-608b35328d0c None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:33.254856 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1753d26f-9ad2-4041-92ab-608b35328d0c None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:33.255515 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1753d26f-9ad2-4041-92ab-608b35328d0c None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:33.274135 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application [None req-1753d26f-9ad2-4041-92ab-608b35328d0c None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 01:43:33.274135 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 01:43:33.274135 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 01:43:33.274135 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 01:43:33.274135 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 01:43:33.274135 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 01:43:33.274135 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 01:43:33.274135 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 01:43:33.274135 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 01:43:33.274135 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 01:43:33.274135 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 01:43:33.274135 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 01:43:33.274135 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 30 01:43:33.274135 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 30 01:43:33.274135 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 30 01:43:33.274135 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 30 01:43:33.275150 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 01:43:33.275150 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 01:43:33.275150 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 01:43:33.275150 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 01:43:33.275150 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 01:43:33.275150 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 01:43:33.275150 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 01:43:33.275150 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application with Lock( Aug 30 01:43:33.275150 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 01:43:33.275150 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._enter() Aug 30 01:43:33.275150 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 01:43:33.275150 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 01:43:33.275150 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 01:43:33.275150 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.creator() Aug 30 01:43:33.275150 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 01:43:33.275150 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application created_value = creator( Aug 30 01:43:33.275150 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 30 01:43:33.276056 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 30 01:43:33.276056 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 01:43:33.276056 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Aug 30 01:43:33.276743 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 339/674] 10.222.0.27 () {56 vars in 1108 bytes} [Tue Aug 30 01:43:33 2022] GET /identity/v3/domains/Default => generated 87 bytes in 202 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 01:43:33.277394 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1753d26f-9ad2-4041-92ab-608b35328d0c None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:33.300920 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-431871fd-de52-4a2f-86a4-36c750400c80 None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:33.342455 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f79940ae-c79f-491b-ad16-f30a92d8c936 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:33.343607 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f79940ae-c79f-491b-ad16-f30a92d8c936 None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:33.344086 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f79940ae-c79f-491b-ad16-f30a92d8c936 None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:33.344546 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f79940ae-c79f-491b-ad16-f30a92d8c936 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:33.347699 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:43:33.354014 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f79940ae-c79f-491b-ad16-f30a92d8c936 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:33.366654 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f79940ae-c79f-491b-ad16-f30a92d8c936 None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:33.367668 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application [None req-f79940ae-c79f-491b-ad16-f30a92d8c936 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 30 01:43:33.367668 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 01:43:33.367668 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 01:43:33.367668 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 01:43:33.367668 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 01:43:33.367668 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 01:43:33.367668 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 01:43:33.367668 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 01:43:33.367668 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 01:43:33.367668 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 01:43:33.367668 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 01:43:33.367668 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 01:43:33.367668 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 30 01:43:33.367668 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 30 01:43:33.367668 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 30 01:43:33.368524 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 30 01:43:33.368524 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 01:43:33.368524 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 01:43:33.368524 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 01:43:33.368524 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 01:43:33.368524 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 01:43:33.368524 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 01:43:33.368524 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 01:43:33.368524 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application with Lock( Aug 30 01:43:33.368524 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 01:43:33.368524 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._enter() Aug 30 01:43:33.368524 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 01:43:33.368524 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 01:43:33.368524 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 01:43:33.368524 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.creator() Aug 30 01:43:33.368524 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 01:43:33.368524 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application created_value = creator( Aug 30 01:43:33.369418 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 30 01:43:33.369418 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 30 01:43:33.369418 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 30 01:43:33.369418 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 30 01:43:33.369418 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 30 01:43:33.369418 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 30 01:43:33.369418 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 30 01:43:33.369418 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application Aug 30 01:43:33.369418 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 336/675] 10.222.0.27 () {56 vars in 1111 bytes} [Tue Aug 30 01:43:33 2022] GET /identity/v3/projects/service => generated 88 bytes in 212 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 01:43:33.369418 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f79940ae-c79f-491b-ad16-f30a92d8c936 None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:33.391332 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:43:33.408616 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1753d26f-9ad2-4041-92ab-608b35328d0c None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:33.445295 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b39b682-9441-4540-9af9-0ccd82028e75 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:33.445982 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b39b682-9441-4540-9af9-0ccd82028e75 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:33.446327 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b39b682-9441-4540-9af9-0ccd82028e75 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:33.446607 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b39b682-9441-4540-9af9-0ccd82028e75 None admin] PATH_INFO: `/v3/domains` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:33.447699 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b39b682-9441-4540-9af9-0ccd82028e75 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:43:33.447699 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b39b682-9441-4540-9af9-0ccd82028e75 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:33.449048 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b39b682-9441-4540-9af9-0ccd82028e75 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:33.462145 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 340/676] 10.222.0.27 () {56 vars in 1109 bytes} [Tue Aug 30 01:43:33 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:43:33.462963 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:43:33.464262 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b39b682-9441-4540-9af9-0ccd82028e75 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:33.518280 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f79940ae-c79f-491b-ad16-f30a92d8c936 None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:33.551744 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:43:33.570461 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a14a74f1-9c45-4379-9660-42a021b6352b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:33.572439 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a14a74f1-9c45-4379-9660-42a021b6352b None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:33.572439 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a14a74f1-9c45-4379-9660-42a021b6352b None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:33.572439 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a14a74f1-9c45-4379-9660-42a021b6352b None admin] PATH_INFO: `/v3/domains` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:33.572940 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a14a74f1-9c45-4379-9660-42a021b6352b None admin] RBAC: Adding query filter params (name=Default) {{(pid=81578) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:43:33.573613 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a14a74f1-9c45-4379-9660-42a021b6352b None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:33.574986 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a14a74f1-9c45-4379-9660-42a021b6352b None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:33.588634 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b39b682-9441-4540-9af9-0ccd82028e75 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:33.592376 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 337/677] 10.222.0.27 () {56 vars in 1109 bytes} [Tue Aug 30 01:43:33 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 223 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:43:33.593226 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a14a74f1-9c45-4379-9660-42a021b6352b None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:33.632286 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c475ad2-4653-4a61-ab9a-71949183941b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:33.632897 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c475ad2-4653-4a61-ab9a-71949183941b None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:33.633177 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c475ad2-4653-4a61-ab9a-71949183941b None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:33.633450 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c475ad2-4653-4a61-ab9a-71949183941b None admin] PATH_INFO: `/v3/projects` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:33.634174 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c475ad2-4653-4a61-ab9a-71949183941b None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:43:33.634480 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c475ad2-4653-4a61-ab9a-71949183941b None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:33.636104 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c475ad2-4653-4a61-ab9a-71949183941b None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:33.651721 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 341/678] 10.222.0.27 () {56 vars in 1148 bytes} [Tue Aug 30 01:43:33 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 189 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:43:33.652590 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c475ad2-4653-4a61-ab9a-71949183941b None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:33.732551 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a14a74f1-9c45-4379-9660-42a021b6352b None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:33.765834 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:43:33.781885 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff80a6bc-9829-45d2-98b0-5a47bda2dca4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:33.782691 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff80a6bc-9829-45d2-98b0-5a47bda2dca4 None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:33.783039 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff80a6bc-9829-45d2-98b0-5a47bda2dca4 None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:33.789584 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff80a6bc-9829-45d2-98b0-5a47bda2dca4 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:33.789724 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c475ad2-4653-4a61-ab9a-71949183941b None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:33.790896 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff80a6bc-9829-45d2-98b0-5a47bda2dca4 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:33.792545 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff80a6bc-9829-45d2-98b0-5a47bda2dca4 None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:33.804382 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application [None req-ff80a6bc-9829-45d2-98b0-5a47bda2dca4 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 30 01:43:33.804382 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 01:43:33.804382 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 01:43:33.804382 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 01:43:33.804382 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 01:43:33.804382 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 01:43:33.804382 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 01:43:33.804382 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 01:43:33.804382 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 01:43:33.804382 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 01:43:33.804382 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 01:43:33.804382 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 01:43:33.804382 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 30 01:43:33.804382 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 30 01:43:33.804382 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 30 01:43:33.805299 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 30 01:43:33.805299 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 01:43:33.805299 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 01:43:33.805299 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 01:43:33.805299 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 01:43:33.805299 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 01:43:33.805299 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 01:43:33.805299 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 01:43:33.805299 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application with Lock( Aug 30 01:43:33.805299 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 01:43:33.805299 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._enter() Aug 30 01:43:33.805299 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 01:43:33.805299 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 01:43:33.805299 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 01:43:33.805299 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.creator() Aug 30 01:43:33.805299 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 01:43:33.805299 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application created_value = creator( Aug 30 01:43:33.806701 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 30 01:43:33.806701 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 30 01:43:33.806701 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 30 01:43:33.806701 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 30 01:43:33.806701 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 30 01:43:33.806701 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 30 01:43:33.806701 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 30 01:43:33.806701 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application Aug 30 01:43:33.809553 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 338/679] 10.222.0.27 () {56 vars in 1111 bytes} [Tue Aug 30 01:43:33 2022] GET /identity/v3/projects/service => generated 88 bytes in 216 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 01:43:33.810563 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff80a6bc-9829-45d2-98b0-5a47bda2dca4 None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:33.823219 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:43:33.831231 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0085495-ad59-48a9-a487-fb3930df4e7f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:33.835405 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0085495-ad59-48a9-a487-fb3930df4e7f None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:33.835405 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0085495-ad59-48a9-a487-fb3930df4e7f None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:33.835405 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0085495-ad59-48a9-a487-fb3930df4e7f None admin] PATH_INFO: `/v3/users/swift` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:33.841906 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0085495-ad59-48a9-a487-fb3930df4e7f None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:33.843409 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0085495-ad59-48a9-a487-fb3930df4e7f None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:33.862065 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application [None req-e0085495-ad59-48a9-a487-fb3930df4e7f None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Aug 30 01:43:33.862065 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 01:43:33.862065 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 01:43:33.862065 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 01:43:33.862065 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 01:43:33.862065 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 01:43:33.862065 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 01:43:33.862065 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 01:43:33.862065 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 01:43:33.862065 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 01:43:33.862065 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 01:43:33.862065 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 01:43:33.862065 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 30 01:43:33.862065 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 30 01:43:33.862065 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 30 01:43:33.862065 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 30 01:43:33.863093 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 01:43:33.863093 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 01:43:33.863093 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 30 01:43:33.863093 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 30 01:43:33.863093 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 30 01:43:33.863093 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 30 01:43:33.863093 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 01:43:33.863093 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 01:43:33.863093 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 01:43:33.863093 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 01:43:33.863093 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 01:43:33.863093 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application with Lock( Aug 30 01:43:33.863093 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 01:43:33.863093 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._enter() Aug 30 01:43:33.863093 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 01:43:33.863093 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 01:43:33.863093 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 01:43:33.863093 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.creator() Aug 30 01:43:33.864164 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 01:43:33.864164 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application created_value = creator( Aug 30 01:43:33.864164 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 30 01:43:33.864164 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 30 01:43:33.864164 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 30 01:43:33.864164 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 30 01:43:33.864164 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 30 01:43:33.864164 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 30 01:43:33.864164 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Aug 30 01:43:33.864164 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Aug 30 01:43:33.864164 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 342/680] 10.222.0.27 () {56 vars in 1096 bytes} [Tue Aug 30 01:43:33 2022] GET /identity/v3/users/swift => generated 83 bytes in 211 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 01:43:33.864791 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0085495-ad59-48a9-a487-fb3930df4e7f None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:33.931455 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff80a6bc-9829-45d2-98b0-5a47bda2dca4 None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:33.966058 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-851e75db-fd62-413b-bc85-01a8e7a55804 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:33.966941 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-851e75db-fd62-413b-bc85-01a8e7a55804 None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:33.967407 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-851e75db-fd62-413b-bc85-01a8e7a55804 None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:33.967860 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-851e75db-fd62-413b-bc85-01a8e7a55804 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:33.968773 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-851e75db-fd62-413b-bc85-01a8e7a55804 None admin] RBAC: Adding query filter params (role.id=aa59b0c9009a480994d9df81a5f8d404, scope.project.id=4562f3951fbe465280073e60a11e44e2, user.id=ab936b3767a8417aa80c8c8f1b84b667) {{(pid=81578) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:43:33.969281 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-851e75db-fd62-413b-bc85-01a8e7a55804 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:33.970726 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-851e75db-fd62-413b-bc85-01a8e7a55804 None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:33.975565 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0085495-ad59-48a9-a487-fb3930df4e7f None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:33.976737 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 339/681] 10.222.0.27 () {56 vars in 1374 bytes} [Tue Aug 30 01:43:33 2022] GET /identity/v3/role_assignments?user.id=ab936b3767a8417aa80c8c8f1b84b667&scope.project.id=4562f3951fbe465280073e60a11e44e2&role.id=aa59b0c9009a480994d9df81a5f8d404 => generated 608 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:43:33.979227 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-851e75db-fd62-413b-bc85-01a8e7a55804 None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:34.016533 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d1c12a1-b464-435a-8a89-30d197171351 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:34.017288 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d1c12a1-b464-435a-8a89-30d197171351 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:34.017689 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d1c12a1-b464-435a-8a89-30d197171351 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:34.018088 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d1c12a1-b464-435a-8a89-30d197171351 None admin] PATH_INFO: `/v3/projects` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:34.018966 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d1c12a1-b464-435a-8a89-30d197171351 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:43:34.019364 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d1c12a1-b464-435a-8a89-30d197171351 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:34.020919 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d1c12a1-b464-435a-8a89-30d197171351 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:34.035896 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 343/682] 10.222.0.27 () {56 vars in 1148 bytes} [Tue Aug 30 01:43:33 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:43:34.037481 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1dbbb5e-00df-4bf8-8579-2672d9317980 None None] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:34.037877 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1dbbb5e-00df-4bf8-8579-2672d9317980 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:34.038247 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1dbbb5e-00df-4bf8-8579-2672d9317980 None None] PATH_INFO: `/` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:34.043750 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 344/683] 10.222.0.27 () {54 vars in 927 bytes} [Tue Aug 30 01:43:34 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 01:43:34.044771 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1dbbb5e-00df-4bf8-8579-2672d9317980 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:34.062780 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:43:34.108117 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:43:34.129759 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-851e75db-fd62-413b-bc85-01a8e7a55804 None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:34.170983 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-772a9d6a-4ef7-4d32-8cdf-315cae434a96 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:34.171619 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-772a9d6a-4ef7-4d32-8cdf-315cae434a96 None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:34.173622 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-772a9d6a-4ef7-4d32-8cdf-315cae434a96 None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:34.173622 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-772a9d6a-4ef7-4d32-8cdf-315cae434a96 None admin] PATH_INFO: `/v3/users` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:34.173846 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-772a9d6a-4ef7-4d32-8cdf-315cae434a96 None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=81578) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:43:34.173970 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-772a9d6a-4ef7-4d32-8cdf-315cae434a96 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:34.175888 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-772a9d6a-4ef7-4d32-8cdf-315cae434a96 None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:34.185355 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1dbbb5e-00df-4bf8-8579-2672d9317980 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:34.196444 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 340/684] 10.222.0.27 () {56 vars in 1135 bytes} [Tue Aug 30 01:43:33 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 219 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:43:34.198603 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bab42c5b-3f5f-4222-86ae-a552cffda2ba None None] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:34.199089 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bab42c5b-3f5f-4222-86ae-a552cffda2ba None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:34.199884 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bab42c5b-3f5f-4222-86ae-a552cffda2ba None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:34.225526 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.auth.core [None req-bab42c5b-3f5f-4222-86ae-a552cffda2ba None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81578) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 01:43:34.227280 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-bab42c5b-3f5f-4222-86ae-a552cffda2ba None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:43:34.233253 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7541b028-281d-4822-a134-63d342ed05aa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:34.234021 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7541b028-281d-4822-a134-63d342ed05aa None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:34.234395 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7541b028-281d-4822-a134-63d342ed05aa None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:34.234760 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7541b028-281d-4822-a134-63d342ed05aa None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:34.235581 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7541b028-281d-4822-a134-63d342ed05aa None admin] RBAC: Adding query filter params (role.id=aa59b0c9009a480994d9df81a5f8d404, scope.project.id=4562f3951fbe465280073e60a11e44e2, user.id=f3ea1580c56f4814a92bbb2acf542905) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:43:34.236053 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7541b028-281d-4822-a134-63d342ed05aa None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:34.238267 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7541b028-281d-4822-a134-63d342ed05aa None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:34.248286 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 345/685] 10.222.0.27 () {56 vars in 1374 bytes} [Tue Aug 30 01:43:34 2022] GET /identity/v3/role_assignments?user.id=f3ea1580c56f4814a92bbb2acf542905&scope.project.id=4562f3951fbe465280073e60a11e44e2&role.id=aa59b0c9009a480994d9df81a5f8d404 => generated 608 bytes in 203 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:43:34.249372 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7541b028-281d-4822-a134-63d342ed05aa None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:34.350946 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 341/686] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Aug 30 01:43:34 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 152 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 30 01:43:34.432539 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7541b028-281d-4822-a134-63d342ed05aa None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:34.446960 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:43:34.468748 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f95c7b06-99de-4644-b20e-c10c9568b979 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:34.469354 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f95c7b06-99de-4644-b20e-c10c9568b979 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:34.469660 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f95c7b06-99de-4644-b20e-c10c9568b979 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:34.469927 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f95c7b06-99de-4644-b20e-c10c9568b979 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:34.540269 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f95c7b06-99de-4644-b20e-c10c9568b979 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:34.541757 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f95c7b06-99de-4644-b20e-c10c9568b979 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:34.650873 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 346/687] 10.222.0.27 () {58 vars in 1282 bytes} [Tue Aug 30 01:43:34 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 402 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 30 01:43:34.656091 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bab42c5b-3f5f-4222-86ae-a552cffda2ba None None] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:34.700763 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f95c7b06-99de-4644-b20e-c10c9568b979 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:34.722626 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:43:34.731021 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:43:34.830619 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f95c7b06-99de-4644-b20e-c10c9568b979 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:34.836891 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bab42c5b-3f5f-4222-86ae-a552cffda2ba None None] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:34.891685 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd4eca63-b532-424a-b435-655593d1bb80 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:34.892330 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd4eca63-b532-424a-b435-655593d1bb80 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:34.892615 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd4eca63-b532-424a-b435-655593d1bb80 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:34.892893 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd4eca63-b532-424a-b435-655593d1bb80 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:34.906799 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd4eca63-b532-424a-b435-655593d1bb80 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:34.908235 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd4eca63-b532-424a-b435-655593d1bb80 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:34.918532 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60ac3fd2-ecea-475e-bc5e-5c2590162e55 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:34.919195 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60ac3fd2-ecea-475e-bc5e-5c2590162e55 None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:34.919507 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60ac3fd2-ecea-475e-bc5e-5c2590162e55 None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:34.919740 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60ac3fd2-ecea-475e-bc5e-5c2590162e55 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:34.933744 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application [None req-dd4eca63-b532-424a-b435-655593d1bb80 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 01:43:34.933744 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 01:43:34.933744 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 01:43:34.933744 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 01:43:34.933744 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 01:43:34.933744 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 01:43:34.933744 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 01:43:34.933744 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 01:43:34.933744 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 01:43:34.933744 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 01:43:34.933744 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 01:43:34.933744 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 01:43:34.933744 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 30 01:43:34.933744 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 30 01:43:34.933744 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 30 01:43:34.933744 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 30 01:43:34.934762 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 01:43:34.934762 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 01:43:34.934762 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 01:43:34.934762 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 01:43:34.934762 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 01:43:34.934762 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 01:43:34.934762 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 01:43:34.934762 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application with Lock( Aug 30 01:43:34.934762 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 01:43:34.934762 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._enter() Aug 30 01:43:34.934762 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 01:43:34.934762 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 01:43:34.934762 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 01:43:34.934762 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.creator() Aug 30 01:43:34.934762 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 01:43:34.934762 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application created_value = creator( Aug 30 01:43:34.934762 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 30 01:43:34.935669 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 30 01:43:34.935669 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 01:43:34.935669 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Aug 30 01:43:34.942232 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 347/688] 10.222.0.27 () {56 vars in 1108 bytes} [Tue Aug 30 01:43:34 2022] GET /identity/v3/domains/Default => generated 87 bytes in 237 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 01:43:34.945715 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd4eca63-b532-424a-b435-655593d1bb80 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:34.956905 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60ac3fd2-ecea-475e-bc5e-5c2590162e55 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:34.958771 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60ac3fd2-ecea-475e-bc5e-5c2590162e55 None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:34.990847 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application [None req-60ac3fd2-ecea-475e-bc5e-5c2590162e55 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 01:43:34.990847 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 01:43:34.990847 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 01:43:34.990847 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 01:43:34.990847 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 01:43:34.990847 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 01:43:34.990847 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 01:43:34.990847 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 01:43:34.990847 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 01:43:34.990847 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 01:43:34.990847 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 01:43:34.990847 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 01:43:34.990847 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 30 01:43:34.990847 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 30 01:43:34.990847 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 30 01:43:34.990847 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 30 01:43:34.991983 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 01:43:34.991983 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 01:43:34.991983 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 01:43:34.991983 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 01:43:34.991983 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 01:43:34.991983 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 01:43:34.991983 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 01:43:34.991983 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application with Lock( Aug 30 01:43:34.991983 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 01:43:34.991983 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._enter() Aug 30 01:43:34.991983 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 01:43:34.991983 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 01:43:34.991983 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 01:43:34.991983 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.creator() Aug 30 01:43:34.991983 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 01:43:34.991983 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application created_value = creator( Aug 30 01:43:34.991983 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 30 01:43:34.993123 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 30 01:43:34.993123 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 01:43:34.993123 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application Aug 30 01:43:34.997493 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 342/689] 10.222.0.27 () {56 vars in 1108 bytes} [Tue Aug 30 01:43:34 2022] GET /identity/v3/domains/Default => generated 87 bytes in 342 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 01:43:34.998325 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60ac3fd2-ecea-475e-bc5e-5c2590162e55 None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:35.045865 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:43:35.110885 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd4eca63-b532-424a-b435-655593d1bb80 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:35.182842 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d56fb56-af9b-4ef2-aadb-5ee5b1777e13 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:35.183435 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d56fb56-af9b-4ef2-aadb-5ee5b1777e13 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:35.183717 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d56fb56-af9b-4ef2-aadb-5ee5b1777e13 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:35.183995 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d56fb56-af9b-4ef2-aadb-5ee5b1777e13 None admin] PATH_INFO: `/v3/domains` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:35.184649 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d56fb56-af9b-4ef2-aadb-5ee5b1777e13 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:43:35.184953 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d56fb56-af9b-4ef2-aadb-5ee5b1777e13 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:35.186365 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d56fb56-af9b-4ef2-aadb-5ee5b1777e13 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:35.199232 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:43:35.211217 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60ac3fd2-ecea-475e-bc5e-5c2590162e55 None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:35.213702 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 348/690] 10.222.0.27 () {56 vars in 1109 bytes} [Tue Aug 30 01:43:34 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 268 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:43:35.216695 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d56fb56-af9b-4ef2-aadb-5ee5b1777e13 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:35.281517 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b8c1f14-3eaf-41d3-ab7b-f28e2d052ab0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:35.282148 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b8c1f14-3eaf-41d3-ab7b-f28e2d052ab0 None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:35.282428 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b8c1f14-3eaf-41d3-ab7b-f28e2d052ab0 None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:35.282699 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b8c1f14-3eaf-41d3-ab7b-f28e2d052ab0 None admin] PATH_INFO: `/v3/domains` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:35.283355 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b8c1f14-3eaf-41d3-ab7b-f28e2d052ab0 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81578) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:43:35.283661 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b8c1f14-3eaf-41d3-ab7b-f28e2d052ab0 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:35.285618 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b8c1f14-3eaf-41d3-ab7b-f28e2d052ab0 None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:35.305743 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 343/691] 10.222.0.27 () {56 vars in 1109 bytes} [Tue Aug 30 01:43:34 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 305 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:43:35.307174 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faa5ae4f-38f9-4604-80bf-448abe512a97 None None] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:35.307526 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faa5ae4f-38f9-4604-80bf-448abe512a97 None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:35.307804 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faa5ae4f-38f9-4604-80bf-448abe512a97 None None] PATH_INFO: `/` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:35.309478 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 344/692] 10.222.0.27 () {54 vars in 927 bytes} [Tue Aug 30 01:43:35 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 01:43:35.310433 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-faa5ae4f-38f9-4604-80bf-448abe512a97 None None] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:35.380457 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d56fb56-af9b-4ef2-aadb-5ee5b1777e13 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:35.416460 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:43:35.426011 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-faa5ae4f-38f9-4604-80bf-448abe512a97 None None] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:35.437776 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4ce8525-d537-4f1f-9603-b28b3772eac9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:35.439154 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4ce8525-d537-4f1f-9603-b28b3772eac9 None admin] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:35.439962 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4ce8525-d537-4f1f-9603-b28b3772eac9 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:35.440454 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4ce8525-d537-4f1f-9603-b28b3772eac9 None admin] PATH_INFO: `/v3/users` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:35.441618 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4ce8525-d537-4f1f-9603-b28b3772eac9 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:35.443373 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4ce8525-d537-4f1f-9603-b28b3772eac9 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:35.474717 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-679deab2-1d30-468e-8129-1b215f3f684a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:35.475684 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-679deab2-1d30-468e-8129-1b215f3f684a None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:35.476023 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-679deab2-1d30-468e-8129-1b215f3f684a None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:35.476549 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-679deab2-1d30-468e-8129-1b215f3f684a None admin] PATH_INFO: `/v3/projects/service` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:35.483718 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-679deab2-1d30-468e-8129-1b215f3f684a None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:35.485218 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-679deab2-1d30-468e-8129-1b215f3f684a None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:35.496965 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application [None req-679deab2-1d30-468e-8129-1b215f3f684a None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 30 01:43:35.496965 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 01:43:35.496965 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 01:43:35.496965 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 01:43:35.496965 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 01:43:35.496965 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 01:43:35.496965 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 01:43:35.496965 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 01:43:35.496965 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 01:43:35.496965 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 01:43:35.496965 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 01:43:35.496965 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 01:43:35.496965 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 30 01:43:35.496965 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 30 01:43:35.496965 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 30 01:43:35.497888 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 30 01:43:35.497888 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 01:43:35.497888 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 01:43:35.497888 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 01:43:35.497888 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 01:43:35.497888 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 01:43:35.497888 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 01:43:35.497888 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 01:43:35.497888 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application with Lock( Aug 30 01:43:35.497888 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 01:43:35.497888 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._enter() Aug 30 01:43:35.497888 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 01:43:35.497888 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 01:43:35.497888 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 01:43:35.497888 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.creator() Aug 30 01:43:35.497888 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 01:43:35.497888 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application created_value = creator( Aug 30 01:43:35.498788 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 30 01:43:35.498788 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 30 01:43:35.498788 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 30 01:43:35.498788 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 30 01:43:35.498788 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 30 01:43:35.498788 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 30 01:43:35.498788 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 30 01:43:35.498788 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application Aug 30 01:43:35.501520 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 345/693] 10.222.0.27 () {56 vars in 1111 bytes} [Tue Aug 30 01:43:35 2022] GET /identity/v3/projects/service => generated 88 bytes in 188 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 01:43:35.503107 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30fa46ee-b84f-4243-b2bd-0749b7a4b010 None None] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:35.503445 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30fa46ee-b84f-4243-b2bd-0749b7a4b010 None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:35.503664 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30fa46ee-b84f-4243-b2bd-0749b7a4b010 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:35.523660 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.auth.core [None req-30fa46ee-b84f-4243-b2bd-0749b7a4b010 None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81578) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 01:43:35.525113 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-30fa46ee-b84f-4243-b2bd-0749b7a4b010 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:43:35.528972 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:43:35.554687 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 349/694] 10.222.0.27 () {60 vars in 1132 bytes} [Tue Aug 30 01:43:35 2022] POST /identity/v3/users => generated 254 bytes in 334 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 01:43:35.555743 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4ce8525-d537-4f1f-9603-b28b3772eac9 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:35.632765 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 346/695] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Aug 30 01:43:35 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 130 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 30 01:43:35.739025 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4ce8525-d537-4f1f-9603-b28b3772eac9 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:35.803884 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af4216c6-6bda-48fe-8a92-479dfcec1874 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:35.804517 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af4216c6-6bda-48fe-8a92-479dfcec1874 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:35.804796 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af4216c6-6bda-48fe-8a92-479dfcec1874 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:35.805067 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af4216c6-6bda-48fe-8a92-479dfcec1874 None admin] PATH_INFO: `/v3/projects` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:35.805820 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af4216c6-6bda-48fe-8a92-479dfcec1874 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:43:35.806131 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af4216c6-6bda-48fe-8a92-479dfcec1874 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:35.807790 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af4216c6-6bda-48fe-8a92-479dfcec1874 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:35.824296 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 350/696] 10.222.0.27 () {56 vars in 1148 bytes} [Tue Aug 30 01:43:35 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 269 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:43:35.831088 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30fa46ee-b84f-4243-b2bd-0749b7a4b010 None None] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:35.956960 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30fa46ee-b84f-4243-b2bd-0749b7a4b010 None None] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:35.998687 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f140619-d892-4e4f-b145-58e6aa1247dc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:35.999659 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f140619-d892-4e4f-b145-58e6aa1247dc None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:36.000438 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f140619-d892-4e4f-b145-58e6aa1247dc None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:36.001105 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f140619-d892-4e4f-b145-58e6aa1247dc None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:36.002078 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f140619-d892-4e4f-b145-58e6aa1247dc None admin] RBAC: Adding query filter params (role.id=aa59b0c9009a480994d9df81a5f8d404, scope.project.id=4562f3951fbe465280073e60a11e44e2, user.id=b4f0008fde424f22bc1f7307cf422774) {{(pid=81578) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:43:36.002602 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f140619-d892-4e4f-b145-58e6aa1247dc None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:36.004213 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f140619-d892-4e4f-b145-58e6aa1247dc None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:36.011979 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 347/697] 10.222.0.27 () {56 vars in 1374 bytes} [Tue Aug 30 01:43:35 2022] GET /identity/v3/role_assignments?user.id=b4f0008fde424f22bc1f7307cf422774&scope.project.id=4562f3951fbe465280073e60a11e44e2&role.id=aa59b0c9009a480994d9df81a5f8d404 => generated 608 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:43:36.019575 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af4216c6-6bda-48fe-8a92-479dfcec1874 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:36.078022 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:43:36.152624 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af4216c6-6bda-48fe-8a92-479dfcec1874 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:36.195222 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0b654be-1444-4c9b-a25a-8248cee78ba0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:36.196013 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0b654be-1444-4c9b-a25a-8248cee78ba0 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:36.196390 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0b654be-1444-4c9b-a25a-8248cee78ba0 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:36.197976 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0b654be-1444-4c9b-a25a-8248cee78ba0 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:36.210722 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0b654be-1444-4c9b-a25a-8248cee78ba0 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 30 01:43:36.217360 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application [None req-e0b654be-1444-4c9b-a25a-8248cee78ba0 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 30 01:43:36.217360 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 01:43:36.217360 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 01:43:36.217360 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 01:43:36.217360 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 01:43:36.217360 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 01:43:36.217360 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 01:43:36.217360 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 01:43:36.217360 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 01:43:36.217360 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 01:43:36.217360 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 01:43:36.217360 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 01:43:36.217360 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 30 01:43:36.217360 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 30 01:43:36.217360 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 30 01:43:36.217360 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 30 01:43:36.218636 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 30 01:43:36.218636 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 30 01:43:36.218636 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 30 01:43:36.218636 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 30 01:43:36.218636 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 01:43:36.218636 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 01:43:36.218636 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 01:43:36.218636 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 01:43:36.218636 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 01:43:36.218636 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 01:43:36.218636 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 01:43:36.218636 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application with Lock( Aug 30 01:43:36.218636 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 01:43:36.218636 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._enter() Aug 30 01:43:36.218636 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 01:43:36.218636 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 01:43:36.219688 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 01:43:36.219688 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.creator() Aug 30 01:43:36.219688 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 01:43:36.219688 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application created_value = creator( Aug 30 01:43:36.219688 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 30 01:43:36.219688 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 30 01:43:36.219688 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 30 01:43:36.219688 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 30 01:43:36.219688 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 30 01:43:36.219688 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 30 01:43:36.219688 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 30 01:43:36.219688 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Aug 30 01:43:36.221913 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 351/698] 10.222.0.27 () {56 vars in 1096 bytes} [Tue Aug 30 01:43:36 2022] GET /identity/v3/roles/admin => generated 83 bytes in 203 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 01:43:36.231853 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f140619-d892-4e4f-b145-58e6aa1247dc None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:36.239525 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:43:36.370756 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f140619-d892-4e4f-b145-58e6aa1247dc None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:36.384146 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecb4bb7b-6cf7-43e4-bf81-c4eb0404f6bf None None] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:36.384449 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecb4bb7b-6cf7-43e4-bf81-c4eb0404f6bf None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:36.384674 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecb4bb7b-6cf7-43e4-bf81-c4eb0404f6bf None None] PATH_INFO: `/` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:36.391558 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 352/699] 10.222.0.27 () {54 vars in 927 bytes} [Tue Aug 30 01:43:36 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 30 01:43:36.393317 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8d425ad-c444-4820-a4a5-06fa08f7b9e5 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:36.393650 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8d425ad-c444-4820-a4a5-06fa08f7b9e5 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:36.393934 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8d425ad-c444-4820-a4a5-06fa08f7b9e5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:36.416625 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-d8d425ad-c444-4820-a4a5-06fa08f7b9e5 None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 01:43:36.417862 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-d8d425ad-c444-4820-a4a5-06fa08f7b9e5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:43:36.418775 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53d9d300-5efa-4b6d-ad95-e3e82d986ce2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:36.419813 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53d9d300-5efa-4b6d-ad95-e3e82d986ce2 None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:36.420317 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53d9d300-5efa-4b6d-ad95-e3e82d986ce2 None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:36.420596 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53d9d300-5efa-4b6d-ad95-e3e82d986ce2 None admin] PATH_INFO: `/v3/roles` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:36.421274 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53d9d300-5efa-4b6d-ad95-e3e82d986ce2 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81578) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:43:36.421609 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53d9d300-5efa-4b6d-ad95-e3e82d986ce2 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:36.423939 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53d9d300-5efa-4b6d-ad95-e3e82d986ce2 None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:36.434866 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 348/700] 10.222.0.27 () {56 vars in 1099 bytes} [Tue Aug 30 01:43:36 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 203 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:43:36.440643 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53d9d300-5efa-4b6d-ad95-e3e82d986ce2 None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:36.509600 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 353/701] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Aug 30 01:43:36 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Aug 30 01:43:36.525873 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:43:36.577344 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53d9d300-5efa-4b6d-ad95-e3e82d986ce2 None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:36.625619 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62fa6d19-b362-4926-993f-90f23adb0a82 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:36.626795 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62fa6d19-b362-4926-993f-90f23adb0a82 None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:36.627435 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62fa6d19-b362-4926-993f-90f23adb0a82 None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:36.627914 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62fa6d19-b362-4926-993f-90f23adb0a82 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:36.643423 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62fa6d19-b362-4926-993f-90f23adb0a82 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:36.646243 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62fa6d19-b362-4926-993f-90f23adb0a82 None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:36.667569 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application [None req-62fa6d19-b362-4926-993f-90f23adb0a82 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 01:43:36.667569 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 01:43:36.667569 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 01:43:36.667569 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 01:43:36.667569 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 01:43:36.667569 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 01:43:36.667569 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 01:43:36.667569 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 01:43:36.667569 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 01:43:36.667569 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 01:43:36.667569 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 01:43:36.667569 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 01:43:36.667569 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 30 01:43:36.667569 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 30 01:43:36.667569 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 30 01:43:36.667569 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 30 01:43:36.669572 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 01:43:36.669572 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 01:43:36.669572 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 01:43:36.669572 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 01:43:36.669572 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 01:43:36.669572 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 01:43:36.669572 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 01:43:36.669572 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application with Lock( Aug 30 01:43:36.669572 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 01:43:36.669572 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._enter() Aug 30 01:43:36.669572 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 01:43:36.669572 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 01:43:36.669572 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 01:43:36.669572 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.creator() Aug 30 01:43:36.669572 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 01:43:36.669572 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application created_value = creator( Aug 30 01:43:36.669572 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 30 01:43:36.670530 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 30 01:43:36.670530 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 01:43:36.670530 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application Aug 30 01:43:36.672913 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 349/702] 10.222.0.27 () {56 vars in 1108 bytes} [Tue Aug 30 01:43:36 2022] GET /identity/v3/domains/Default => generated 87 bytes in 233 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 01:43:36.679520 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8d425ad-c444-4820-a4a5-06fa08f7b9e5 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:36.726099 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:43:36.748404 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b906cded-27c2-4f76-8f33-72ef1fc8c92d None None] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:36.748735 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b906cded-27c2-4f76-8f33-72ef1fc8c92d None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:36.748944 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b906cded-27c2-4f76-8f33-72ef1fc8c92d None None] PATH_INFO: `/` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:36.750466 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 350/703] 10.222.0.27 () {54 vars in 927 bytes} [Tue Aug 30 01:43:36 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 01:43:36.756707 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de576739-2dbe-44ef-8837-256f76d507d6 None None] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:36.757722 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de576739-2dbe-44ef-8837-256f76d507d6 None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:36.758196 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de576739-2dbe-44ef-8837-256f76d507d6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:36.778006 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.auth.core [None req-de576739-2dbe-44ef-8837-256f76d507d6 None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81578) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 01:43:36.779627 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-de576739-2dbe-44ef-8837-256f76d507d6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:43:36.852820 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8d425ad-c444-4820-a4a5-06fa08f7b9e5 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:36.872163 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 351/704] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Aug 30 01:43:36 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 117 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Aug 30 01:43:36.878359 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de576739-2dbe-44ef-8837-256f76d507d6 None None] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:36.921652 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee787bd9-f21c-4250-8a1e-2ca9ac82239f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:36.922371 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee787bd9-f21c-4250-8a1e-2ca9ac82239f None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:36.922722 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee787bd9-f21c-4250-8a1e-2ca9ac82239f None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:36.922995 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee787bd9-f21c-4250-8a1e-2ca9ac82239f None admin] PATH_INFO: `/v3/domains` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:36.923664 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee787bd9-f21c-4250-8a1e-2ca9ac82239f None admin] RBAC: Adding query filter params (name=Default) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:43:36.924023 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee787bd9-f21c-4250-8a1e-2ca9ac82239f None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:36.927151 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee787bd9-f21c-4250-8a1e-2ca9ac82239f None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:36.947058 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 354/705] 10.222.0.27 () {56 vars in 1109 bytes} [Tue Aug 30 01:43:36 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 268 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:43:36.953892 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee787bd9-f21c-4250-8a1e-2ca9ac82239f None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:36.977105 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:43:37.054426 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de576739-2dbe-44ef-8837-256f76d507d6 None None] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:37.116128 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:43:37.123409 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee787bd9-f21c-4250-8a1e-2ca9ac82239f None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:37.150303 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b972af5e-9b3a-4a76-90d3-a308d69dac0d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:37.150303 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b972af5e-9b3a-4a76-90d3-a308d69dac0d None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:37.150303 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b972af5e-9b3a-4a76-90d3-a308d69dac0d None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:37.150303 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b972af5e-9b3a-4a76-90d3-a308d69dac0d None admin] PATH_INFO: `/v3/services/block-storage` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:37.150665 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b972af5e-9b3a-4a76-90d3-a308d69dac0d None admin] RBAC: Authorizing `identity:get_service(service_id=block-storage)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:37.150665 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b972af5e-9b3a-4a76-90d3-a308d69dac0d None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:37.169545 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application [None req-b972af5e-9b3a-4a76-90d3-a308d69dac0d None admin] Could not find service: block-storage.: keystone.exception.ServiceNotFound: Could not find service: block-storage. Aug 30 01:43:37.169545 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 01:43:37.169545 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Aug 30 01:43:37.169545 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Aug 30 01:43:37.169545 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Aug 30 01:43:37.169545 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Aug 30 01:43:37.169545 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Aug 30 01:43:37.169545 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 30 01:43:37.169545 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Aug 30 01:43:37.169545 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application Aug 30 01:43:37.169545 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 30 01:43:37.169545 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application Aug 30 01:43:37.169545 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 01:43:37.169545 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 01:43:37.169545 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 01:43:37.169545 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 01:43:37.169545 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 01:43:37.171026 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 01:43:37.171026 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 01:43:37.171026 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 01:43:37.171026 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 01:43:37.171026 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 01:43:37.171026 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 01:43:37.171026 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Aug 30 01:43:37.171026 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._get_service(service_id) Aug 30 01:43:37.171026 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Aug 30 01:43:37.171026 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Aug 30 01:43:37.171026 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 01:43:37.171026 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 01:43:37.171026 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 01:43:37.171026 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 01:43:37.171026 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 01:43:37.171026 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 01:43:37.171886 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 01:43:37.171886 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application with Lock( Aug 30 01:43:37.171886 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 01:43:37.171886 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._enter() Aug 30 01:43:37.171886 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 01:43:37.171886 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 01:43:37.171886 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 01:43:37.171886 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.creator() Aug 30 01:43:37.171886 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 01:43:37.171886 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application created_value = creator( Aug 30 01:43:37.171886 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Aug 30 01:43:37.171886 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 30 01:43:37.171886 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Aug 30 01:43:37.171886 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application Aug 30 01:43:37.171886 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 352/706] 10.222.0.27 () {56 vars in 1129 bytes} [Tue Aug 30 01:43:36 2022] GET /identity/v3/services/block-storage => generated 94 bytes in 297 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 01:43:37.179572 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b972af5e-9b3a-4a76-90d3-a308d69dac0d None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:37.179836 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5315e98b-897d-470b-987e-a70f3f32d35b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:37.180474 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5315e98b-897d-470b-987e-a70f3f32d35b None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:37.180752 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5315e98b-897d-470b-987e-a70f3f32d35b None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:37.181321 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5315e98b-897d-470b-987e-a70f3f32d35b None admin] PATH_INFO: `/v3/users/nova` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:37.197510 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5315e98b-897d-470b-987e-a70f3f32d35b None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:37.199299 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5315e98b-897d-470b-987e-a70f3f32d35b None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:37.215216 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application [None req-5315e98b-897d-470b-987e-a70f3f32d35b None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Aug 30 01:43:37.215216 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 01:43:37.215216 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 01:43:37.215216 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 01:43:37.215216 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 01:43:37.215216 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 01:43:37.215216 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 01:43:37.215216 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 01:43:37.215216 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 01:43:37.215216 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 01:43:37.215216 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 01:43:37.215216 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 01:43:37.215216 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 30 01:43:37.215216 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 30 01:43:37.215216 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 30 01:43:37.215216 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 30 01:43:37.217362 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 01:43:37.217362 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 01:43:37.217362 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 30 01:43:37.217362 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 30 01:43:37.217362 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 30 01:43:37.217362 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 30 01:43:37.217362 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 01:43:37.217362 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 01:43:37.217362 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 01:43:37.217362 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 01:43:37.217362 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 01:43:37.217362 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application with Lock( Aug 30 01:43:37.217362 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 01:43:37.217362 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._enter() Aug 30 01:43:37.217362 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 01:43:37.217362 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 01:43:37.217362 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 01:43:37.217362 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.creator() Aug 30 01:43:37.219581 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 01:43:37.219581 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application created_value = creator( Aug 30 01:43:37.219581 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 30 01:43:37.219581 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 30 01:43:37.219581 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 30 01:43:37.219581 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 30 01:43:37.219581 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 30 01:43:37.219581 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 30 01:43:37.219581 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Aug 30 01:43:37.219581 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Aug 30 01:43:37.219581 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 355/707] 10.222.0.27 () {56 vars in 1093 bytes} [Tue Aug 30 01:43:36 2022] GET /identity/v3/users/nova => generated 82 bytes in 264 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 01:43:37.219581 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5315e98b-897d-470b-987e-a70f3f32d35b None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:37.318175 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b972af5e-9b3a-4a76-90d3-a308d69dac0d None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:37.333855 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:43:37.341430 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5315e98b-897d-470b-987e-a70f3f32d35b None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:37.364703 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e722b3fe-f43a-464b-904c-443d888c1bec None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:37.365758 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e722b3fe-f43a-464b-904c-443d888c1bec None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:37.366101 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e722b3fe-f43a-464b-904c-443d888c1bec None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:37.366441 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e722b3fe-f43a-464b-904c-443d888c1bec None admin] PATH_INFO: `/v3/services` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:37.367385 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e722b3fe-f43a-464b-904c-443d888c1bec None admin] RBAC: Adding query filter params (name=block-storage) {{(pid=81578) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:43:37.367751 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e722b3fe-f43a-464b-904c-443d888c1bec None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:37.369967 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e722b3fe-f43a-464b-904c-443d888c1bec None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:37.376810 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 353/708] 10.222.0.27 () {56 vars in 1124 bytes} [Tue Aug 30 01:43:37 2022] GET /identity/v3/services?name=block-storage => generated 130 bytes in 197 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:43:37.377729 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e722b3fe-f43a-464b-904c-443d888c1bec None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:37.380154 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ea96f47-e52b-42e1-b729-55dd44f49474 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:37.381630 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ea96f47-e52b-42e1-b729-55dd44f49474 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:37.382265 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ea96f47-e52b-42e1-b729-55dd44f49474 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:37.383015 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ea96f47-e52b-42e1-b729-55dd44f49474 None admin] PATH_INFO: `/v3/services/network` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:37.383919 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ea96f47-e52b-42e1-b729-55dd44f49474 None admin] RBAC: Authorizing `identity:get_service(service_id=network)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:37.385656 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ea96f47-e52b-42e1-b729-55dd44f49474 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:37.400232 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application [None req-2ea96f47-e52b-42e1-b729-55dd44f49474 None admin] Could not find service: network.: keystone.exception.ServiceNotFound: Could not find service: network. Aug 30 01:43:37.400232 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 01:43:37.400232 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Aug 30 01:43:37.400232 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Aug 30 01:43:37.400232 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Aug 30 01:43:37.400232 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Aug 30 01:43:37.400232 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Aug 30 01:43:37.400232 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 30 01:43:37.400232 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Aug 30 01:43:37.400232 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Aug 30 01:43:37.400232 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 30 01:43:37.400232 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Aug 30 01:43:37.400232 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 01:43:37.400232 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 01:43:37.400232 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 01:43:37.400232 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 01:43:37.400232 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 01:43:37.401350 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 01:43:37.401350 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 01:43:37.401350 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 01:43:37.401350 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 01:43:37.401350 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 01:43:37.401350 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 01:43:37.401350 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Aug 30 01:43:37.401350 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_service(service_id) Aug 30 01:43:37.401350 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Aug 30 01:43:37.401350 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Aug 30 01:43:37.401350 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 01:43:37.401350 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 01:43:37.401350 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 01:43:37.401350 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 01:43:37.401350 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 01:43:37.401350 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 01:43:37.402564 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 01:43:37.402564 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application with Lock( Aug 30 01:43:37.402564 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 01:43:37.402564 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._enter() Aug 30 01:43:37.402564 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 01:43:37.402564 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 01:43:37.402564 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 01:43:37.402564 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.creator() Aug 30 01:43:37.402564 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 01:43:37.402564 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application created_value = creator( Aug 30 01:43:37.402564 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Aug 30 01:43:37.402564 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 30 01:43:37.402564 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Aug 30 01:43:37.402564 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Aug 30 01:43:37.405153 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 356/709] 10.222.0.27 () {56 vars in 1111 bytes} [Tue Aug 30 01:43:37 2022] GET /identity/v3/services/network => generated 88 bytes in 188 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 01:43:37.406200 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ea96f47-e52b-42e1-b729-55dd44f49474 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:37.425426 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:43:37.514511 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ea96f47-e52b-42e1-b729-55dd44f49474 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:37.516235 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e722b3fe-f43a-464b-904c-443d888c1bec None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:37.550849 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-688bb0c8-68ed-4765-a0e6-72281caeabb2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:37.551854 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-688bb0c8-68ed-4765-a0e6-72281caeabb2 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:37.552339 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-688bb0c8-68ed-4765-a0e6-72281caeabb2 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:37.553356 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3666bb66-c2e1-4580-9d95-dbee87e9bf4e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:37.553522 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-688bb0c8-68ed-4765-a0e6-72281caeabb2 None admin] PATH_INFO: `/v3/services` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:37.554279 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-688bb0c8-68ed-4765-a0e6-72281caeabb2 None admin] RBAC: Adding query filter params (type=block-storage) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:43:37.554391 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3666bb66-c2e1-4580-9d95-dbee87e9bf4e None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:37.554769 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3666bb66-c2e1-4580-9d95-dbee87e9bf4e None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:37.554977 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-688bb0c8-68ed-4765-a0e6-72281caeabb2 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:37.555214 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3666bb66-c2e1-4580-9d95-dbee87e9bf4e None admin] PATH_INFO: `/v3/users` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:37.556031 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3666bb66-c2e1-4580-9d95-dbee87e9bf4e None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81578) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:43:37.556432 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3666bb66-c2e1-4580-9d95-dbee87e9bf4e None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:37.556989 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-688bb0c8-68ed-4765-a0e6-72281caeabb2 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:37.558029 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3666bb66-c2e1-4580-9d95-dbee87e9bf4e None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:37.563425 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 357/710] 10.222.0.27 () {56 vars in 1124 bytes} [Tue Aug 30 01:43:37 2022] GET /identity/v3/services?type=block-storage => generated 130 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:43:37.564460 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-688bb0c8-68ed-4765-a0e6-72281caeabb2 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:37.578922 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 354/711] 10.222.0.27 () {56 vars in 1133 bytes} [Tue Aug 30 01:43:37 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 201 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:43:37.585610 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3666bb66-c2e1-4580-9d95-dbee87e9bf4e None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:37.624827 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:43:37.686907 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:43:37.735139 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3666bb66-c2e1-4580-9d95-dbee87e9bf4e None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:37.738981 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-688bb0c8-68ed-4765-a0e6-72281caeabb2 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:37.793783 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fada18f-7618-44c8-9c1e-ac56b3127f89 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:37.795497 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fada18f-7618-44c8-9c1e-ac56b3127f89 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:37.796298 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fada18f-7618-44c8-9c1e-ac56b3127f89 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:37.796789 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fada18f-7618-44c8-9c1e-ac56b3127f89 None admin] PATH_INFO: `/v3/services` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:37.797962 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fada18f-7618-44c8-9c1e-ac56b3127f89 None admin] RBAC: Adding query filter params (name=network) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:43:37.798490 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fada18f-7618-44c8-9c1e-ac56b3127f89 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:37.800715 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fada18f-7618-44c8-9c1e-ac56b3127f89 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:37.807158 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 358/712] 10.222.0.27 () {56 vars in 1112 bytes} [Tue Aug 30 01:43:37 2022] GET /identity/v3/services?name=network => generated 124 bytes in 243 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:43:37.811604 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed6c08af-eff2-4ee2-aa27-b69fc1f27547 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:37.811604 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed6c08af-eff2-4ee2-aa27-b69fc1f27547 None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:37.811604 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed6c08af-eff2-4ee2-aa27-b69fc1f27547 None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:37.811604 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed6c08af-eff2-4ee2-aa27-b69fc1f27547 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:37.814537 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fada18f-7618-44c8-9c1e-ac56b3127f89 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:37.906291 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed6c08af-eff2-4ee2-aa27-b69fc1f27547 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:37.908368 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed6c08af-eff2-4ee2-aa27-b69fc1f27547 None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:37.946684 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fada18f-7618-44c8-9c1e-ac56b3127f89 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:37.951788 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:43:37.983387 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a38c0795-ff46-4662-883e-6e21e6b5bfa4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:37.984020 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a38c0795-ff46-4662-883e-6e21e6b5bfa4 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:37.984361 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a38c0795-ff46-4662-883e-6e21e6b5bfa4 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:37.984641 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a38c0795-ff46-4662-883e-6e21e6b5bfa4 None admin] PATH_INFO: `/v3/services` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:37.985305 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a38c0795-ff46-4662-883e-6e21e6b5bfa4 None admin] RBAC: Adding query filter params (type=network) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:43:37.985640 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a38c0795-ff46-4662-883e-6e21e6b5bfa4 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:37.986976 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a38c0795-ff46-4662-883e-6e21e6b5bfa4 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:37.992051 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 359/713] 10.222.0.27 () {56 vars in 1112 bytes} [Tue Aug 30 01:43:37 2022] GET /identity/v3/services?type=network => generated 124 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:43:38.028637 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 355/714] 10.222.0.27 () {58 vars in 1282 bytes} [Tue Aug 30 01:43:37 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 444 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 30 01:43:38.034556 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a38c0795-ff46-4662-883e-6e21e6b5bfa4 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:38.047029 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:43:38.165703 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a38c0795-ff46-4662-883e-6e21e6b5bfa4 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:38.202094 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96b50699-76af-4dc8-aac4-e4a400db2555 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:38.202837 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96b50699-76af-4dc8-aac4-e4a400db2555 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:38.203210 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96b50699-76af-4dc8-aac4-e4a400db2555 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:38.203580 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96b50699-76af-4dc8-aac4-e4a400db2555 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:38.216455 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96b50699-76af-4dc8-aac4-e4a400db2555 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:38.218126 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96b50699-76af-4dc8-aac4-e4a400db2555 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:38.234677 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application [None req-96b50699-76af-4dc8-aac4-e4a400db2555 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 01:43:38.234677 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 01:43:38.234677 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 01:43:38.234677 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 01:43:38.234677 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 01:43:38.234677 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 01:43:38.234677 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 01:43:38.234677 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 01:43:38.234677 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 01:43:38.234677 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 01:43:38.234677 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 01:43:38.234677 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 01:43:38.234677 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 30 01:43:38.234677 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 30 01:43:38.234677 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 30 01:43:38.234677 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 30 01:43:38.235944 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 01:43:38.235944 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 01:43:38.235944 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 01:43:38.235944 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 01:43:38.235944 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 01:43:38.235944 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 01:43:38.235944 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 01:43:38.235944 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application with Lock( Aug 30 01:43:38.235944 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 01:43:38.235944 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._enter() Aug 30 01:43:38.235944 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 01:43:38.235944 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 01:43:38.235944 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 01:43:38.235944 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.creator() Aug 30 01:43:38.235944 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 01:43:38.235944 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application created_value = creator( Aug 30 01:43:38.235944 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 30 01:43:38.237066 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 30 01:43:38.237066 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 01:43:38.237066 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Aug 30 01:43:38.238405 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 360/715] 10.222.0.27 () {56 vars in 1108 bytes} [Tue Aug 30 01:43:38 2022] GET /identity/v3/domains/Default => generated 87 bytes in 204 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 01:43:38.243734 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed6c08af-eff2-4ee2-aa27-b69fc1f27547 None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:38.321999 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2d8080f-c20e-4638-887e-322009c10f55 None None] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:38.322505 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2d8080f-c20e-4638-887e-322009c10f55 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:38.322878 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2d8080f-c20e-4638-887e-322009c10f55 None None] PATH_INFO: `/` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:38.324543 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 361/716] 10.222.0.27 () {54 vars in 927 bytes} [Tue Aug 30 01:43:38 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 30 01:43:38.330011 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d4a7a54-a2d7-4d20-ba42-db4762a6b34b None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:38.330405 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d4a7a54-a2d7-4d20-ba42-db4762a6b34b None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:38.330775 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d4a7a54-a2d7-4d20-ba42-db4762a6b34b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:38.352864 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-1d4a7a54-a2d7-4d20-ba42-db4762a6b34b None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 01:43:38.354237 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-1d4a7a54-a2d7-4d20-ba42-db4762a6b34b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:43:38.355296 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed6c08af-eff2-4ee2-aa27-b69fc1f27547 None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:38.397484 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8935f71-f840-48ea-a374-12b47a87ae34 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:38.397484 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8935f71-f840-48ea-a374-12b47a87ae34 None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:38.397484 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8935f71-f840-48ea-a374-12b47a87ae34 None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:38.397484 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8935f71-f840-48ea-a374-12b47a87ae34 None admin] PATH_INFO: `/v3/domains` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:38.397891 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8935f71-f840-48ea-a374-12b47a87ae34 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81578) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:43:38.397891 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8935f71-f840-48ea-a374-12b47a87ae34 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:38.399225 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8935f71-f840-48ea-a374-12b47a87ae34 None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:38.400865 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:43:38.412063 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 356/717] 10.222.0.27 () {56 vars in 1109 bytes} [Tue Aug 30 01:43:38 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 30 01:43:38.418428 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8935f71-f840-48ea-a374-12b47a87ae34 None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:38.445171 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:43:38.470658 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 362/718] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Aug 30 01:43:38 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 141 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Aug 30 01:43:38.544334 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8935f71-f840-48ea-a374-12b47a87ae34 None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:38.580106 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c52af2c1-6bc0-4394-8f74-d1355291cf86 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:38.580732 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c52af2c1-6bc0-4394-8f74-d1355291cf86 None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:38.581006 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c52af2c1-6bc0-4394-8f74-d1355291cf86 None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:38.581271 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c52af2c1-6bc0-4394-8f74-d1355291cf86 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:38.588179 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c52af2c1-6bc0-4394-8f74-d1355291cf86 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:38.589713 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c52af2c1-6bc0-4394-8f74-d1355291cf86 None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:38.601605 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application [None req-c52af2c1-6bc0-4394-8f74-d1355291cf86 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 30 01:43:38.601605 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 01:43:38.601605 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 01:43:38.601605 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 01:43:38.601605 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 01:43:38.601605 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 01:43:38.601605 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 01:43:38.601605 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 01:43:38.601605 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 01:43:38.601605 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 01:43:38.601605 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 01:43:38.601605 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 01:43:38.601605 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 30 01:43:38.601605 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 30 01:43:38.601605 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 30 01:43:38.602507 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 30 01:43:38.602507 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 01:43:38.602507 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 01:43:38.602507 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 01:43:38.602507 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 01:43:38.602507 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 01:43:38.602507 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 01:43:38.602507 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 01:43:38.602507 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application with Lock( Aug 30 01:43:38.602507 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 01:43:38.602507 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._enter() Aug 30 01:43:38.602507 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 01:43:38.602507 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 01:43:38.602507 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 01:43:38.602507 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.creator() Aug 30 01:43:38.602507 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 01:43:38.602507 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application created_value = creator( Aug 30 01:43:38.603399 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 30 01:43:38.603399 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 30 01:43:38.603399 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 30 01:43:38.603399 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 30 01:43:38.603399 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 30 01:43:38.603399 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 30 01:43:38.603399 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 30 01:43:38.603399 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application Aug 30 01:43:38.603399 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 357/719] 10.222.0.27 () {56 vars in 1111 bytes} [Tue Aug 30 01:43:38 2022] GET /identity/v3/projects/service => generated 88 bytes in 185 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 30 01:43:38.609940 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d4a7a54-a2d7-4d20-ba42-db4762a6b34b None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:38.672436 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7085a765-08b6-4f5e-9a44-6d92329011a3 None None] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:38.672749 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7085a765-08b6-4f5e-9a44-6d92329011a3 None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:38.672972 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7085a765-08b6-4f5e-9a44-6d92329011a3 None None] PATH_INFO: `/` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:38.674693 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 358/720] 10.222.0.27 () {54 vars in 927 bytes} [Tue Aug 30 01:43:38 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 01:43:38.681659 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-483cb998-fb69-4467-bb2b-3902628a1de0 None None] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:38.681938 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-483cb998-fb69-4467-bb2b-3902628a1de0 None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:38.682229 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-483cb998-fb69-4467-bb2b-3902628a1de0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:38.735044 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.auth.core [None req-483cb998-fb69-4467-bb2b-3902628a1de0 None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81578) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 01:43:38.736470 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-483cb998-fb69-4467-bb2b-3902628a1de0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:43:38.768803 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d4a7a54-a2d7-4d20-ba42-db4762a6b34b None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:38.827487 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:43:38.841636 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 359/721] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Aug 30 01:43:38 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 161 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Aug 30 01:43:38.842528 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b70f216d-5bd8-4230-8b49-d74b608f9ac3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:38.843260 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b70f216d-5bd8-4230-8b49-d74b608f9ac3 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:38.843616 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b70f216d-5bd8-4230-8b49-d74b608f9ac3 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:38.843886 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b70f216d-5bd8-4230-8b49-d74b608f9ac3 None admin] PATH_INFO: `/v3/projects` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:38.844580 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b70f216d-5bd8-4230-8b49-d74b608f9ac3 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:43:38.844882 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b70f216d-5bd8-4230-8b49-d74b608f9ac3 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:38.846356 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b70f216d-5bd8-4230-8b49-d74b608f9ac3 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:38.867550 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 363/722] 10.222.0.27 () {56 vars in 1148 bytes} [Tue Aug 30 01:43:38 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 258 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:43:38.873436 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-483cb998-fb69-4467-bb2b-3902628a1de0 None None] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:38.913409 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b70f216d-5bd8-4230-8b49-d74b608f9ac3 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:39.019288 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:43:39.064189 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-483cb998-fb69-4467-bb2b-3902628a1de0 None None] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:39.102298 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec81b13f-80e8-4ac8-adae-4f9e0b2d1d62 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:39.102905 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec81b13f-80e8-4ac8-adae-4f9e0b2d1d62 None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:39.103186 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec81b13f-80e8-4ac8-adae-4f9e0b2d1d62 None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:39.103453 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec81b13f-80e8-4ac8-adae-4f9e0b2d1d62 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:39.104179 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec81b13f-80e8-4ac8-adae-4f9e0b2d1d62 None admin] RBAC: Adding query filter params (role.id=ead5fac86a0747ea8823a82efcea8ed7, scope.project.id=4562f3951fbe465280073e60a11e44e2, user.id=d64f2e71361644589bec8a06e6eceed1) {{(pid=81578) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:43:39.104490 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec81b13f-80e8-4ac8-adae-4f9e0b2d1d62 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:39.105938 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec81b13f-80e8-4ac8-adae-4f9e0b2d1d62 None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:39.112614 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 360/723] 10.222.0.27 () {56 vars in 1374 bytes} [Tue Aug 30 01:43:38 2022] GET /identity/v3/role_assignments?user.id=d64f2e71361644589bec8a06e6eceed1&scope.project.id=4562f3951fbe465280073e60a11e44e2&role.id=ead5fac86a0747ea8823a82efcea8ed7 => generated 259 bytes in 239 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:43:39.145981 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b70f216d-5bd8-4230-8b49-d74b608f9ac3 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:39.173449 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec81b13f-80e8-4ac8-adae-4f9e0b2d1d62 None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:39.219096 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb4da730-8901-46b3-9d45-d85401d9aaa3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:39.219789 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb4da730-8901-46b3-9d45-d85401d9aaa3 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:39.220008 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb4da730-8901-46b3-9d45-d85401d9aaa3 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:39.220332 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb4da730-8901-46b3-9d45-d85401d9aaa3 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:39.234958 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:43:39.258479 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb4da730-8901-46b3-9d45-d85401d9aaa3 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 30 01:43:39.264141 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application [None req-cb4da730-8901-46b3-9d45-d85401d9aaa3 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Aug 30 01:43:39.264141 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 01:43:39.264141 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 01:43:39.264141 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 01:43:39.264141 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 01:43:39.264141 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 01:43:39.264141 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 01:43:39.264141 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 01:43:39.264141 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 01:43:39.264141 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 01:43:39.264141 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 01:43:39.264141 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 01:43:39.264141 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 30 01:43:39.264141 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 30 01:43:39.264141 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 30 01:43:39.264141 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 30 01:43:39.265077 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 30 01:43:39.265077 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 30 01:43:39.265077 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 30 01:43:39.265077 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 30 01:43:39.265077 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 01:43:39.265077 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 01:43:39.265077 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 01:43:39.265077 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 01:43:39.265077 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 01:43:39.265077 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 01:43:39.265077 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 01:43:39.265077 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application with Lock( Aug 30 01:43:39.265077 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 01:43:39.265077 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._enter() Aug 30 01:43:39.265077 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 01:43:39.265077 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 01:43:39.265955 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 01:43:39.265955 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.creator() Aug 30 01:43:39.265955 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 01:43:39.265955 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application created_value = creator( Aug 30 01:43:39.265955 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 30 01:43:39.265955 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 30 01:43:39.265955 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 30 01:43:39.265955 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 30 01:43:39.265955 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 30 01:43:39.265955 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 30 01:43:39.265955 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Aug 30 01:43:39.265955 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Aug 30 01:43:39.269917 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 364/724] 10.222.0.27 () {56 vars in 1102 bytes} [Tue Aug 30 01:43:38 2022] GET /identity/v3/roles/service => generated 85 bytes in 357 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 01:43:39.274958 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb4da730-8901-46b3-9d45-d85401d9aaa3 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:39.291547 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec81b13f-80e8-4ac8-adae-4f9e0b2d1d62 None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:39.336090 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1096c35e-9374-41ce-bbd3-363f384c23e6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:39.337132 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1096c35e-9374-41ce-bbd3-363f384c23e6 None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:39.337679 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1096c35e-9374-41ce-bbd3-363f384c23e6 None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:39.338177 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1096c35e-9374-41ce-bbd3-363f384c23e6 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:39.357019 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1096c35e-9374-41ce-bbd3-363f384c23e6 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 30 01:43:39.363794 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application [None req-1096c35e-9374-41ce-bbd3-363f384c23e6 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 30 01:43:39.363794 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 01:43:39.363794 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 01:43:39.363794 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 01:43:39.363794 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 01:43:39.363794 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 01:43:39.363794 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 01:43:39.363794 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 01:43:39.363794 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 01:43:39.363794 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 01:43:39.363794 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 01:43:39.363794 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 01:43:39.363794 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 30 01:43:39.363794 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 30 01:43:39.363794 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 30 01:43:39.363794 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 30 01:43:39.364830 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 30 01:43:39.364830 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 30 01:43:39.364830 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 30 01:43:39.364830 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 30 01:43:39.364830 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 01:43:39.364830 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 01:43:39.364830 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 01:43:39.364830 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 01:43:39.364830 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 01:43:39.364830 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 01:43:39.364830 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 01:43:39.364830 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application with Lock( Aug 30 01:43:39.364830 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 01:43:39.364830 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._enter() Aug 30 01:43:39.364830 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 01:43:39.364830 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 01:43:39.370585 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 01:43:39.370585 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.creator() Aug 30 01:43:39.370585 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 01:43:39.370585 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application created_value = creator( Aug 30 01:43:39.370585 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 30 01:43:39.370585 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 30 01:43:39.370585 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 30 01:43:39.370585 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 30 01:43:39.370585 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 30 01:43:39.370585 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 30 01:43:39.370585 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 30 01:43:39.370585 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application Aug 30 01:43:39.370585 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 361/725] 10.222.0.27 () {56 vars in 1096 bytes} [Tue Aug 30 01:43:39 2022] GET /identity/v3/roles/admin => generated 83 bytes in 193 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 30 01:43:39.373731 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1096c35e-9374-41ce-bbd3-363f384c23e6 None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:39.460347 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb4da730-8901-46b3-9d45-d85401d9aaa3 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:39.493842 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:43:39.514148 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02816245-e0ff-4bb0-8d76-bb703da846c6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:39.515663 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02816245-e0ff-4bb0-8d76-bb703da846c6 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:39.516245 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02816245-e0ff-4bb0-8d76-bb703da846c6 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:39.518138 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02816245-e0ff-4bb0-8d76-bb703da846c6 None admin] PATH_INFO: `/v3/roles` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:39.518973 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02816245-e0ff-4bb0-8d76-bb703da846c6 None admin] RBAC: Adding query filter params (name=service) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:43:39.523048 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02816245-e0ff-4bb0-8d76-bb703da846c6 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:39.523048 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02816245-e0ff-4bb0-8d76-bb703da846c6 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:39.540408 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 365/726] 10.222.0.27 () {56 vars in 1103 bytes} [Tue Aug 30 01:43:39 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 265 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 30 01:43:39.543541 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02816245-e0ff-4bb0-8d76-bb703da846c6 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:39.556676 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1096c35e-9374-41ce-bbd3-363f384c23e6 None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:39.570084 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:43:39.620124 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbec5f02-b989-4b30-baf3-f761ca76fca9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:39.620729 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbec5f02-b989-4b30-baf3-f761ca76fca9 None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:39.621003 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbec5f02-b989-4b30-baf3-f761ca76fca9 None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:39.621265 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbec5f02-b989-4b30-baf3-f761ca76fca9 None admin] PATH_INFO: `/v3/roles` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:39.621960 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbec5f02-b989-4b30-baf3-f761ca76fca9 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81578) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:43:39.622514 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbec5f02-b989-4b30-baf3-f761ca76fca9 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:39.625034 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbec5f02-b989-4b30-baf3-f761ca76fca9 None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:39.634306 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 362/727] 10.222.0.27 () {56 vars in 1099 bytes} [Tue Aug 30 01:43:39 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 261 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 30 01:43:39.642724 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbec5f02-b989-4b30-baf3-f761ca76fca9 None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:39.696340 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02816245-e0ff-4bb0-8d76-bb703da846c6 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:39.740393 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58cc5ec3-4044-4a29-b8d6-99513972744d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:39.741582 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58cc5ec3-4044-4a29-b8d6-99513972744d None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:39.742069 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58cc5ec3-4044-4a29-b8d6-99513972744d None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:39.742532 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58cc5ec3-4044-4a29-b8d6-99513972744d None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:39.763778 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58cc5ec3-4044-4a29-b8d6-99513972744d None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:39.766549 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58cc5ec3-4044-4a29-b8d6-99513972744d None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:39.793517 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application [None req-58cc5ec3-4044-4a29-b8d6-99513972744d None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 01:43:39.793517 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 01:43:39.793517 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 01:43:39.793517 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 01:43:39.793517 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 01:43:39.793517 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 01:43:39.793517 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 01:43:39.793517 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 01:43:39.793517 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 01:43:39.793517 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 01:43:39.793517 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 01:43:39.793517 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 01:43:39.793517 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 30 01:43:39.793517 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 30 01:43:39.793517 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 30 01:43:39.793517 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 30 01:43:39.794504 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 01:43:39.794504 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 01:43:39.794504 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 01:43:39.794504 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 01:43:39.794504 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 01:43:39.794504 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 01:43:39.794504 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 01:43:39.794504 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application with Lock( Aug 30 01:43:39.794504 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 01:43:39.794504 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._enter() Aug 30 01:43:39.794504 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 01:43:39.794504 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 01:43:39.794504 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 01:43:39.794504 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.creator() Aug 30 01:43:39.794504 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 01:43:39.794504 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application created_value = creator( Aug 30 01:43:39.794504 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 30 01:43:39.795413 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 30 01:43:39.795413 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 01:43:39.795413 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Aug 30 01:43:39.801125 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 366/728] 10.222.0.27 () {56 vars in 1108 bytes} [Tue Aug 30 01:43:39 2022] GET /identity/v3/domains/Default => generated 87 bytes in 258 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 30 01:43:39.819630 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58cc5ec3-4044-4a29-b8d6-99513972744d None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:39.842942 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbec5f02-b989-4b30-baf3-f761ca76fca9 None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:39.870126 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:43:39.886145 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30ec4e5d-f0ee-41f4-9405-e88f59b8b1cb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:39.886789 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30ec4e5d-f0ee-41f4-9405-e88f59b8b1cb None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:39.887068 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30ec4e5d-f0ee-41f4-9405-e88f59b8b1cb None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:39.887340 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30ec4e5d-f0ee-41f4-9405-e88f59b8b1cb None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:39.911389 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30ec4e5d-f0ee-41f4-9405-e88f59b8b1cb None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:39.913692 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30ec4e5d-f0ee-41f4-9405-e88f59b8b1cb None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:39.939629 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application [None req-30ec4e5d-f0ee-41f4-9405-e88f59b8b1cb None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 01:43:39.939629 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 01:43:39.939629 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 01:43:39.939629 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 01:43:39.939629 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 01:43:39.939629 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 01:43:39.939629 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 01:43:39.939629 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 01:43:39.939629 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 01:43:39.939629 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 01:43:39.939629 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 01:43:39.939629 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 01:43:39.939629 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 30 01:43:39.939629 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 30 01:43:39.939629 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 30 01:43:39.939629 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 30 01:43:39.940638 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 01:43:39.940638 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 01:43:39.940638 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 01:43:39.940638 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 01:43:39.940638 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 01:43:39.940638 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 01:43:39.940638 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 01:43:39.940638 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application with Lock( Aug 30 01:43:39.940638 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 01:43:39.940638 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._enter() Aug 30 01:43:39.940638 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 01:43:39.940638 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 01:43:39.940638 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 01:43:39.940638 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.creator() Aug 30 01:43:39.940638 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 01:43:39.940638 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application created_value = creator( Aug 30 01:43:39.940638 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 30 01:43:39.941602 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 30 01:43:39.941602 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 01:43:39.941602 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application Aug 30 01:43:39.941602 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 363/729] 10.222.0.27 () {56 vars in 1108 bytes} [Tue Aug 30 01:43:39 2022] GET /identity/v3/domains/Default => generated 87 bytes in 299 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 01:43:39.948157 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30ec4e5d-f0ee-41f4-9405-e88f59b8b1cb None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:39.957149 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58cc5ec3-4044-4a29-b8d6-99513972744d None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:39.996906 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a55fabae-73e4-4b90-ad43-72e9d1fe896d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:39.997558 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a55fabae-73e4-4b90-ad43-72e9d1fe896d None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:39.997998 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a55fabae-73e4-4b90-ad43-72e9d1fe896d None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:39.998280 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a55fabae-73e4-4b90-ad43-72e9d1fe896d None admin] PATH_INFO: `/v3/domains` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:39.999251 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a55fabae-73e4-4b90-ad43-72e9d1fe896d None admin] RBAC: Adding query filter params (name=Default) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:43:39.999569 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a55fabae-73e4-4b90-ad43-72e9d1fe896d None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:40.001044 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a55fabae-73e4-4b90-ad43-72e9d1fe896d None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:40.015288 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:43:40.015477 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 367/730] 10.222.0.27 () {56 vars in 1109 bytes} [Tue Aug 30 01:43:39 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 197 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 30 01:43:40.020930 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a55fabae-73e4-4b90-ad43-72e9d1fe896d None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:40.121763 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:43:40.135849 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a55fabae-73e4-4b90-ad43-72e9d1fe896d None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:40.174047 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30ec4e5d-f0ee-41f4-9405-e88f59b8b1cb None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:40.174656 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05c93d35-2c8c-4c5f-8894-83f976306a96 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:40.175223 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05c93d35-2c8c-4c5f-8894-83f976306a96 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:40.175598 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05c93d35-2c8c-4c5f-8894-83f976306a96 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:40.176026 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05c93d35-2c8c-4c5f-8894-83f976306a96 None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:40.184297 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05c93d35-2c8c-4c5f-8894-83f976306a96 None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:40.185905 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05c93d35-2c8c-4c5f-8894-83f976306a96 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:40.197985 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application [None req-05c93d35-2c8c-4c5f-8894-83f976306a96 None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Aug 30 01:43:40.197985 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 01:43:40.197985 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 01:43:40.197985 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 01:43:40.197985 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 01:43:40.197985 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 01:43:40.197985 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 01:43:40.197985 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 01:43:40.197985 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 01:43:40.197985 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 01:43:40.197985 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 01:43:40.197985 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 01:43:40.197985 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 30 01:43:40.197985 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 30 01:43:40.197985 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 30 01:43:40.197985 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 30 01:43:40.198977 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 01:43:40.198977 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 01:43:40.198977 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 30 01:43:40.198977 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 30 01:43:40.198977 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 30 01:43:40.198977 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 30 01:43:40.198977 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 01:43:40.198977 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 01:43:40.198977 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 01:43:40.198977 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 01:43:40.198977 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 01:43:40.198977 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application with Lock( Aug 30 01:43:40.198977 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 01:43:40.198977 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._enter() Aug 30 01:43:40.198977 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 01:43:40.198977 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 01:43:40.198977 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 01:43:40.198977 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.creator() Aug 30 01:43:40.199947 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 01:43:40.199947 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application created_value = creator( Aug 30 01:43:40.199947 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 30 01:43:40.199947 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 30 01:43:40.199947 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 30 01:43:40.199947 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 30 01:43:40.199947 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 30 01:43:40.199947 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 30 01:43:40.199947 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Aug 30 01:43:40.199947 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Aug 30 01:43:40.199947 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 368/731] 10.222.0.27 () {56 vars in 1117 bytes} [Tue Aug 30 01:43:40 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 179 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 01:43:40.200536 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c529a667-f4aa-40e1-9cb5-f0894272eb2f None None] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:40.205362 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c529a667-f4aa-40e1-9cb5-f0894272eb2f None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:40.205362 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c529a667-f4aa-40e1-9cb5-f0894272eb2f None None] PATH_INFO: `/` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:40.207500 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 369/732] 10.222.0.27 () {54 vars in 927 bytes} [Tue Aug 30 01:43:40 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 01:43:40.209534 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d82a0a9f-c42b-4689-a5b8-f747875b849d None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:40.210297 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d82a0a9f-c42b-4689-a5b8-f747875b849d None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:40.210297 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d82a0a9f-c42b-4689-a5b8-f747875b849d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:40.222994 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41e67134-8cc0-429f-8775-3f4f6d7fd12c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:40.223847 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41e67134-8cc0-429f-8775-3f4f6d7fd12c None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:40.224466 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41e67134-8cc0-429f-8775-3f4f6d7fd12c None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:40.225555 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41e67134-8cc0-429f-8775-3f4f6d7fd12c None admin] PATH_INFO: `/v3/domains` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:40.226637 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41e67134-8cc0-429f-8775-3f4f6d7fd12c None admin] RBAC: Adding query filter params (name=Default) {{(pid=81578) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:43:40.227117 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41e67134-8cc0-429f-8775-3f4f6d7fd12c None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:40.228742 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41e67134-8cc0-429f-8775-3f4f6d7fd12c None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:40.229885 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-d82a0a9f-c42b-4689-a5b8-f747875b849d None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 01:43:40.231228 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-d82a0a9f-c42b-4689-a5b8-f747875b849d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:43:40.243966 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 364/733] 10.222.0.27 () {56 vars in 1109 bytes} [Tue Aug 30 01:43:39 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 296 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:43:40.245065 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41e67134-8cc0-429f-8775-3f4f6d7fd12c None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:40.317713 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 370/734] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Aug 30 01:43:40 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 30 01:43:40.319505 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d82a0a9f-c42b-4689-a5b8-f747875b849d None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:40.338826 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:43:40.355654 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41e67134-8cc0-429f-8775-3f4f6d7fd12c None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:40.392900 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fde664a8-4627-43da-ba71-40120199cf54 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:40.393909 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fde664a8-4627-43da-ba71-40120199cf54 None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:40.394463 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fde664a8-4627-43da-ba71-40120199cf54 None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:40.395008 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fde664a8-4627-43da-ba71-40120199cf54 None admin] PATH_INFO: `/v3/users` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:40.395915 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fde664a8-4627-43da-ba71-40120199cf54 None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=81578) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:43:40.397219 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fde664a8-4627-43da-ba71-40120199cf54 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:40.399069 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fde664a8-4627-43da-ba71-40120199cf54 None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:40.409701 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:43:40.420161 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 365/735] 10.222.0.27 () {56 vars in 1149 bytes} [Tue Aug 30 01:43:40 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:43:40.426052 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9361341c-2191-44e0-afee-9eab4b43bcd6 None None] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:40.426408 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9361341c-2191-44e0-afee-9eab4b43bcd6 None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:40.426753 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9361341c-2191-44e0-afee-9eab4b43bcd6 None None] PATH_INFO: `/` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:40.429288 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 366/736] 10.222.0.27 () {54 vars in 927 bytes} [Tue Aug 30 01:43:40 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 01:43:40.430111 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9361341c-2191-44e0-afee-9eab4b43bcd6 None None] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:40.447574 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d82a0a9f-c42b-4689-a5b8-f747875b849d None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:40.484610 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4b99c0e-a11e-47d0-a309-f6fd83124c50 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:40.485248 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4b99c0e-a11e-47d0-a309-f6fd83124c50 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:40.485552 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4b99c0e-a11e-47d0-a309-f6fd83124c50 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:40.485822 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4b99c0e-a11e-47d0-a309-f6fd83124c50 None admin] PATH_INFO: `/v3/users/swift` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:40.494780 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4b99c0e-a11e-47d0-a309-f6fd83124c50 None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:40.496227 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4b99c0e-a11e-47d0-a309-f6fd83124c50 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:40.509225 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application [None req-b4b99c0e-a11e-47d0-a309-f6fd83124c50 None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Aug 30 01:43:40.509225 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 01:43:40.509225 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 01:43:40.509225 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 01:43:40.509225 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 01:43:40.509225 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 01:43:40.509225 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 01:43:40.509225 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 01:43:40.509225 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 01:43:40.509225 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 01:43:40.509225 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 01:43:40.509225 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 01:43:40.509225 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 30 01:43:40.509225 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 30 01:43:40.509225 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 30 01:43:40.509225 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 30 01:43:40.510266 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 01:43:40.510266 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 01:43:40.510266 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 30 01:43:40.510266 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 30 01:43:40.510266 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 30 01:43:40.510266 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 30 01:43:40.510266 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 01:43:40.510266 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 01:43:40.510266 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 01:43:40.510266 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 01:43:40.510266 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 01:43:40.510266 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application with Lock( Aug 30 01:43:40.510266 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 01:43:40.510266 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._enter() Aug 30 01:43:40.510266 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 01:43:40.510266 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 01:43:40.510266 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 01:43:40.510266 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.creator() Aug 30 01:43:40.511239 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 01:43:40.511239 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application created_value = creator( Aug 30 01:43:40.511239 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 30 01:43:40.511239 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 30 01:43:40.511239 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 30 01:43:40.511239 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 30 01:43:40.511239 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 30 01:43:40.511239 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 30 01:43:40.511239 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Aug 30 01:43:40.511239 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Aug 30 01:43:40.511239 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 371/737] 10.222.0.27 () {56 vars in 1096 bytes} [Tue Aug 30 01:43:40 2022] GET /identity/v3/users/swift => generated 83 bytes in 192 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 01:43:40.512353 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19430521-aca9-4cf5-b7d8-596be63c8899 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:40.512353 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19430521-aca9-4cf5-b7d8-596be63c8899 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:40.512584 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19430521-aca9-4cf5-b7d8-596be63c8899 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:40.530657 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-19430521-aca9-4cf5-b7d8-596be63c8899 None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 01:43:40.531892 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-19430521-aca9-4cf5-b7d8-596be63c8899 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:43:40.550298 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9361341c-2191-44e0-afee-9eab4b43bcd6 None None] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:40.588755 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b305212e-9091-4880-899d-5a6005c59224 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:40.589920 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b305212e-9091-4880-899d-5a6005c59224 None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:40.590506 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b305212e-9091-4880-899d-5a6005c59224 None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:40.590978 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b305212e-9091-4880-899d-5a6005c59224 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:40.609589 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 372/738] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Aug 30 01:43:40 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 30 01:43:40.610492 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19430521-aca9-4cf5-b7d8-596be63c8899 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:40.617822 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:43:40.667810 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b305212e-9091-4880-899d-5a6005c59224 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:40.669732 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b305212e-9091-4880-899d-5a6005c59224 None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:40.678147 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:43:40.738673 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19430521-aca9-4cf5-b7d8-596be63c8899 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:40.783113 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7487f4d6-1509-4b1f-8b10-94fad72e7a71 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:40.784074 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7487f4d6-1509-4b1f-8b10-94fad72e7a71 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:40.784555 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7487f4d6-1509-4b1f-8b10-94fad72e7a71 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:40.785113 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7487f4d6-1509-4b1f-8b10-94fad72e7a71 None admin] PATH_INFO: `/v3/users` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:40.786129 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7487f4d6-1509-4b1f-8b10-94fad72e7a71 None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:43:40.786638 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7487f4d6-1509-4b1f-8b10-94fad72e7a71 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:40.789070 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7487f4d6-1509-4b1f-8b10-94fad72e7a71 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:40.793550 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 367/739] 10.222.0.27 () {58 vars in 1282 bytes} [Tue Aug 30 01:43:40 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 356 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 30 01:43:40.794468 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b305212e-9091-4880-899d-5a6005c59224 None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:40.808268 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 373/740] 10.222.0.27 () {56 vars in 1135 bytes} [Tue Aug 30 01:43:40 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 198 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:43:40.809630 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7487f4d6-1509-4b1f-8b10-94fad72e7a71 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:40.873857 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:43:40.921169 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b305212e-9091-4880-899d-5a6005c59224 None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:40.921628 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7487f4d6-1509-4b1f-8b10-94fad72e7a71 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:40.948264 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:43:40.957906 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4909b80b-8058-44bb-a24b-364d51312834 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:40.958511 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4909b80b-8058-44bb-a24b-364d51312834 None admin] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:40.958907 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4909b80b-8058-44bb-a24b-364d51312834 None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:40.959096 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4909b80b-8058-44bb-a24b-364d51312834 None admin] PATH_INFO: `/v3/services` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:40.959850 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4909b80b-8058-44bb-a24b-364d51312834 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:40.961176 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4909b80b-8058-44bb-a24b-364d51312834 None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:40.964250 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a899a65f-b8dc-441c-91d6-674ce4b92d77 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:40.965191 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a899a65f-b8dc-441c-91d6-674ce4b92d77 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:40.965694 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a899a65f-b8dc-441c-91d6-674ce4b92d77 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:40.966136 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a899a65f-b8dc-441c-91d6-674ce4b92d77 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:40.978223 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a899a65f-b8dc-441c-91d6-674ce4b92d77 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:40.981273 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a899a65f-b8dc-441c-91d6-674ce4b92d77 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:40.998440 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application [None req-a899a65f-b8dc-441c-91d6-674ce4b92d77 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 01:43:40.998440 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 01:43:40.998440 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 01:43:40.998440 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 01:43:40.998440 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 01:43:40.998440 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 01:43:40.998440 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 01:43:40.998440 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 01:43:40.998440 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 01:43:40.998440 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 01:43:40.998440 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 01:43:40.998440 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 01:43:40.998440 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 30 01:43:40.998440 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 30 01:43:40.998440 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 30 01:43:40.998440 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 30 01:43:40.999470 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 01:43:40.999470 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 01:43:40.999470 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 01:43:40.999470 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 01:43:40.999470 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 01:43:40.999470 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 01:43:40.999470 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 01:43:40.999470 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application with Lock( Aug 30 01:43:40.999470 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 01:43:40.999470 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._enter() Aug 30 01:43:40.999470 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 01:43:40.999470 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 01:43:40.999470 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 01:43:40.999470 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.creator() Aug 30 01:43:40.999470 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 01:43:40.999470 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application created_value = creator( Aug 30 01:43:40.999470 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 30 01:43:41.000725 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 30 01:43:41.000725 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 01:43:41.000725 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Aug 30 01:43:41.000725 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 374/741] 10.222.0.27 () {56 vars in 1108 bytes} [Tue Aug 30 01:43:40 2022] GET /identity/v3/domains/Default => generated 87 bytes in 190 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 01:43:41.000725 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a899a65f-b8dc-441c-91d6-674ce4b92d77 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:41.073533 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 368/742] 10.222.0.27 () {60 vars in 1141 bytes} [Tue Aug 30 01:43:40 2022] POST /identity/v3/services => generated 251 bytes in 277 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 01:43:41.073533 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4909b80b-8058-44bb-a24b-364d51312834 None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:41.123104 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a899a65f-b8dc-441c-91d6-674ce4b92d77 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:41.168918 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb314dff-dcb4-4606-bbfa-816b6b0e5e8a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:41.169674 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb314dff-dcb4-4606-bbfa-816b6b0e5e8a None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:41.170021 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb314dff-dcb4-4606-bbfa-816b6b0e5e8a None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:41.170361 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb314dff-dcb4-4606-bbfa-816b6b0e5e8a None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:41.198445 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:43:41.203551 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4909b80b-8058-44bb-a24b-364d51312834 None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:41.211430 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:43:41.244513 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad1945e5-293f-4f20-be47-8dc022c5acb1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:41.245122 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad1945e5-293f-4f20-be47-8dc022c5acb1 None admin] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:41.245401 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad1945e5-293f-4f20-be47-8dc022c5acb1 None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:41.245712 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad1945e5-293f-4f20-be47-8dc022c5acb1 None admin] PATH_INFO: `/v3/services` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:41.246361 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad1945e5-293f-4f20-be47-8dc022c5acb1 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:41.247717 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad1945e5-293f-4f20-be47-8dc022c5acb1 None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:41.249022 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb314dff-dcb4-4606-bbfa-816b6b0e5e8a None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:41.250764 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb314dff-dcb4-4606-bbfa-816b6b0e5e8a None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:41.320262 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 369/743] 10.222.0.27 () {60 vars in 1141 bytes} [Tue Aug 30 01:43:41 2022] POST /identity/v3/services => generated 240 bytes in 248 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 01:43:41.321747 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad1945e5-293f-4f20-be47-8dc022c5acb1 None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:41.379863 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 375/744] 10.222.0.27 () {58 vars in 1282 bytes} [Tue Aug 30 01:43:40 2022] GET /identity/v3/auth/tokens => generated 903 bytes in 380 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 30 01:43:41.382812 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59283a3d-2d68-43f0-80d4-c77652935749 None None] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:41.383245 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59283a3d-2d68-43f0-80d4-c77652935749 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:41.383684 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59283a3d-2d68-43f0-80d4-c77652935749 None None] PATH_INFO: `/` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:41.386232 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 376/745] 10.222.0.27 () {54 vars in 927 bytes} [Tue Aug 30 01:43:41 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 01:43:41.387583 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59283a3d-2d68-43f0-80d4-c77652935749 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:41.481427 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:43:41.492297 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad1945e5-293f-4f20-be47-8dc022c5acb1 None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:41.537346 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59283a3d-2d68-43f0-80d4-c77652935749 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:41.554161 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28f83861-4795-4043-863b-1607926cc08b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:41.555127 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28f83861-4795-4043-863b-1607926cc08b None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:41.555413 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28f83861-4795-4043-863b-1607926cc08b None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:41.555713 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28f83861-4795-4043-863b-1607926cc08b None admin] PATH_INFO: `/v3/domains` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:41.556493 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28f83861-4795-4043-863b-1607926cc08b None admin] RBAC: Adding query filter params (name=Default) {{(pid=81578) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:43:41.556915 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28f83861-4795-4043-863b-1607926cc08b None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:41.558357 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28f83861-4795-4043-863b-1607926cc08b None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:41.577490 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d45c60a6-491b-42a2-8a56-ec784f5a93b2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:41.577490 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d45c60a6-491b-42a2-8a56-ec784f5a93b2 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:41.577490 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d45c60a6-491b-42a2-8a56-ec784f5a93b2 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:41.577490 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d45c60a6-491b-42a2-8a56-ec784f5a93b2 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:41.579707 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 370/746] 10.222.0.27 () {56 vars in 1109 bytes} [Tue Aug 30 01:43:41 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 259 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:43:41.580982 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b9aa74a-b172-46aa-a1d9-f417ff047b89 None None] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:41.581274 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b9aa74a-b172-46aa-a1d9-f417ff047b89 None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:41.581582 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b9aa74a-b172-46aa-a1d9-f417ff047b89 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:41.594743 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d45c60a6-491b-42a2-8a56-ec784f5a93b2 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:41.596898 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d45c60a6-491b-42a2-8a56-ec784f5a93b2 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:41.600997 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.auth.core [None req-4b9aa74a-b172-46aa-a1d9-f417ff047b89 None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81578) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 01:43:41.602588 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-4b9aa74a-b172-46aa-a1d9-f417ff047b89 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:43:41.613765 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application [None req-d45c60a6-491b-42a2-8a56-ec784f5a93b2 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 01:43:41.613765 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 01:43:41.613765 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 01:43:41.613765 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 01:43:41.613765 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 01:43:41.613765 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 01:43:41.613765 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 01:43:41.613765 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 01:43:41.613765 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 01:43:41.613765 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 01:43:41.613765 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 01:43:41.613765 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 01:43:41.613765 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 30 01:43:41.613765 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 30 01:43:41.613765 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 30 01:43:41.613765 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 30 01:43:41.615482 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 01:43:41.615482 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 01:43:41.615482 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 01:43:41.615482 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 01:43:41.615482 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 01:43:41.615482 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 01:43:41.615482 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 01:43:41.615482 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application with Lock( Aug 30 01:43:41.615482 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 01:43:41.615482 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._enter() Aug 30 01:43:41.615482 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 01:43:41.615482 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 01:43:41.615482 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 01:43:41.615482 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.creator() Aug 30 01:43:41.615482 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 01:43:41.615482 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application created_value = creator( Aug 30 01:43:41.615482 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 30 01:43:41.616485 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 30 01:43:41.616485 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 01:43:41.616485 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Aug 30 01:43:41.616485 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 377/747] 10.222.0.27 () {56 vars in 1108 bytes} [Tue Aug 30 01:43:41 2022] GET /identity/v3/domains/Default => generated 87 bytes in 229 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 01:43:41.617617 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:43:41.617726 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d45c60a6-491b-42a2-8a56-ec784f5a93b2 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:41.684793 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 371/748] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Aug 30 01:43:41 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 30 01:43:41.685828 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b9aa74a-b172-46aa-a1d9-f417ff047b89 None None] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:41.762326 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d45c60a6-491b-42a2-8a56-ec784f5a93b2 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:41.784540 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:43:41.801111 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-230481ce-749e-4736-85b2-b690675affbe None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:41.802622 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-230481ce-749e-4736-85b2-b690675affbe None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:41.802622 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-230481ce-749e-4736-85b2-b690675affbe None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:41.803232 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-230481ce-749e-4736-85b2-b690675affbe None admin] PATH_INFO: `/v3/projects/service` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:41.807172 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b9aa74a-b172-46aa-a1d9-f417ff047b89 None None] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:41.811183 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-230481ce-749e-4736-85b2-b690675affbe None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:41.812853 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-230481ce-749e-4736-85b2-b690675affbe None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:41.826899 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application [None req-230481ce-749e-4736-85b2-b690675affbe None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 30 01:43:41.826899 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 01:43:41.826899 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 01:43:41.826899 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 01:43:41.826899 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 01:43:41.826899 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 01:43:41.826899 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 01:43:41.826899 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 01:43:41.826899 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 01:43:41.826899 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 01:43:41.826899 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 01:43:41.826899 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 01:43:41.826899 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 30 01:43:41.826899 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 30 01:43:41.826899 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 30 01:43:41.827843 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 30 01:43:41.827843 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 01:43:41.827843 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 01:43:41.827843 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 01:43:41.827843 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 01:43:41.827843 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 01:43:41.827843 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 01:43:41.827843 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 01:43:41.827843 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application with Lock( Aug 30 01:43:41.827843 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 01:43:41.827843 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._enter() Aug 30 01:43:41.827843 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 01:43:41.827843 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 01:43:41.827843 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 01:43:41.827843 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.creator() Aug 30 01:43:41.827843 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 01:43:41.827843 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application created_value = creator( Aug 30 01:43:41.828744 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 30 01:43:41.828744 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 30 01:43:41.828744 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 30 01:43:41.828744 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 30 01:43:41.828744 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 30 01:43:41.828744 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 30 01:43:41.828744 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 30 01:43:41.828744 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Aug 30 01:43:41.836444 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 378/749] 10.222.0.27 () {56 vars in 1111 bytes} [Tue Aug 30 01:43:41 2022] GET /identity/v3/projects/service => generated 88 bytes in 214 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 01:43:41.841001 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-230481ce-749e-4736-85b2-b690675affbe None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:41.846963 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba739639-31c7-48f8-b663-52c53f176318 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:41.847593 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba739639-31c7-48f8-b663-52c53f176318 None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:41.847883 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba739639-31c7-48f8-b663-52c53f176318 None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:41.848157 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba739639-31c7-48f8-b663-52c53f176318 None admin] PATH_INFO: `/v3/domains` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:41.848781 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba739639-31c7-48f8-b663-52c53f176318 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81578) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:43:41.849151 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba739639-31c7-48f8-b663-52c53f176318 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:41.851080 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba739639-31c7-48f8-b663-52c53f176318 None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:41.875437 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 372/750] 10.222.0.27 () {56 vars in 1109 bytes} [Tue Aug 30 01:43:41 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 190 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:43:41.883372 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba739639-31c7-48f8-b663-52c53f176318 None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:41.894552 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:43:42.002848 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-230481ce-749e-4736-85b2-b690675affbe None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:42.007717 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba739639-31c7-48f8-b663-52c53f176318 None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:42.041409 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f62f16f9-2951-44d4-a639-634569b71341 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:42.042411 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f62f16f9-2951-44d4-a639-634569b71341 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:42.042900 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f62f16f9-2951-44d4-a639-634569b71341 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:42.043378 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f62f16f9-2951-44d4-a639-634569b71341 None admin] PATH_INFO: `/v3/projects` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:42.044269 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f62f16f9-2951-44d4-a639-634569b71341 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:43:42.044794 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f62f16f9-2951-44d4-a639-634569b71341 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:42.046559 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f62f16f9-2951-44d4-a639-634569b71341 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:42.065492 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c417911-b58b-4d6e-b52f-b90b440b0236 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:42.065492 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c417911-b58b-4d6e-b52f-b90b440b0236 None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:42.065492 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c417911-b58b-4d6e-b52f-b90b440b0236 None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:42.065492 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c417911-b58b-4d6e-b52f-b90b440b0236 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:42.065888 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 379/751] 10.222.0.27 () {56 vars in 1148 bytes} [Tue Aug 30 01:43:41 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 221 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 30 01:43:42.066388 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f62f16f9-2951-44d4-a639-634569b71341 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:42.077514 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c417911-b58b-4d6e-b52f-b90b440b0236 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:42.080114 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c417911-b58b-4d6e-b52f-b90b440b0236 None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:42.097094 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application [None req-8c417911-b58b-4d6e-b52f-b90b440b0236 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 30 01:43:42.097094 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 01:43:42.097094 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 01:43:42.097094 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 01:43:42.097094 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 01:43:42.097094 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 01:43:42.097094 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 01:43:42.097094 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 01:43:42.097094 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 01:43:42.097094 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 01:43:42.097094 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 01:43:42.097094 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 01:43:42.097094 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 30 01:43:42.097094 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 30 01:43:42.097094 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 30 01:43:42.098058 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 30 01:43:42.098058 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 01:43:42.098058 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 01:43:42.098058 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 01:43:42.098058 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 01:43:42.098058 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 01:43:42.098058 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 01:43:42.098058 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 01:43:42.098058 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application with Lock( Aug 30 01:43:42.098058 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 01:43:42.098058 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._enter() Aug 30 01:43:42.098058 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 01:43:42.098058 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 01:43:42.098058 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 01:43:42.098058 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.creator() Aug 30 01:43:42.098058 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 01:43:42.098058 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application created_value = creator( Aug 30 01:43:42.099200 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 30 01:43:42.099200 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 30 01:43:42.099200 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 30 01:43:42.099200 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 30 01:43:42.099200 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 30 01:43:42.099200 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 30 01:43:42.099200 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 30 01:43:42.099200 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application Aug 30 01:43:42.101798 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 373/752] 10.222.0.27 () {56 vars in 1111 bytes} [Tue Aug 30 01:43:41 2022] GET /identity/v3/projects/service => generated 88 bytes in 219 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 30 01:43:42.107366 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c417911-b58b-4d6e-b52f-b90b440b0236 None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:42.114313 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:43:42.181934 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:43:42.189433 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f62f16f9-2951-44d4-a639-634569b71341 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:42.222768 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c417911-b58b-4d6e-b52f-b90b440b0236 None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:42.246841 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32106826-edda-4386-af34-106df0d8059c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:42.247481 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32106826-edda-4386-af34-106df0d8059c None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:42.247852 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32106826-edda-4386-af34-106df0d8059c None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:42.248421 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32106826-edda-4386-af34-106df0d8059c None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:42.265621 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32106826-edda-4386-af34-106df0d8059c None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 30 01:43:42.272674 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application [None req-32106826-edda-4386-af34-106df0d8059c None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 30 01:43:42.272674 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 01:43:42.272674 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 01:43:42.272674 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 01:43:42.272674 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 01:43:42.272674 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 01:43:42.272674 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 01:43:42.272674 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 01:43:42.272674 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 01:43:42.272674 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 01:43:42.272674 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 01:43:42.272674 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 01:43:42.272674 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 30 01:43:42.272674 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 30 01:43:42.272674 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 30 01:43:42.272674 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 30 01:43:42.273667 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 30 01:43:42.273667 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 30 01:43:42.273667 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 30 01:43:42.273667 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 30 01:43:42.273667 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 01:43:42.273667 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 01:43:42.273667 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 01:43:42.273667 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 01:43:42.273667 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 01:43:42.273667 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 01:43:42.273667 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 01:43:42.273667 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application with Lock( Aug 30 01:43:42.273667 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 01:43:42.273667 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._enter() Aug 30 01:43:42.273667 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 01:43:42.273667 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 01:43:42.274691 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 01:43:42.274691 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.creator() Aug 30 01:43:42.274691 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 01:43:42.274691 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application created_value = creator( Aug 30 01:43:42.274691 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 30 01:43:42.274691 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 30 01:43:42.274691 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 30 01:43:42.274691 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 30 01:43:42.274691 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 30 01:43:42.274691 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 30 01:43:42.274691 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 30 01:43:42.274691 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Aug 30 01:43:42.279129 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 380/753] 10.222.0.27 () {56 vars in 1096 bytes} [Tue Aug 30 01:43:42 2022] GET /identity/v3/roles/admin => generated 83 bytes in 213 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 01:43:42.279850 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-074b12ce-fcac-42ef-bc3a-66453f8da7c2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:42.280325 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32106826-edda-4386-af34-106df0d8059c None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:42.281024 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-074b12ce-fcac-42ef-bc3a-66453f8da7c2 None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:42.281532 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-074b12ce-fcac-42ef-bc3a-66453f8da7c2 None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:42.282007 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-074b12ce-fcac-42ef-bc3a-66453f8da7c2 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:42.283107 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-074b12ce-fcac-42ef-bc3a-66453f8da7c2 None admin] RBAC: Adding query filter params (role.id=aa59b0c9009a480994d9df81a5f8d404, scope.project.id=4562f3951fbe465280073e60a11e44e2, user.id=121c374d49a64335ab74a431cfbc1320) {{(pid=81578) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:43:42.283644 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-074b12ce-fcac-42ef-bc3a-66453f8da7c2 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:42.285270 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-074b12ce-fcac-42ef-bc3a-66453f8da7c2 None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:42.300409 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 374/754] 10.222.0.27 () {56 vars in 1374 bytes} [Tue Aug 30 01:43:42 2022] GET /identity/v3/role_assignments?user.id=121c374d49a64335ab74a431cfbc1320&scope.project.id=4562f3951fbe465280073e60a11e44e2&role.id=aa59b0c9009a480994d9df81a5f8d404 => generated 259 bytes in 188 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:43:42.300409 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-074b12ce-fcac-42ef-bc3a-66453f8da7c2 None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:42.447898 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-074b12ce-fcac-42ef-bc3a-66453f8da7c2 None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:42.487572 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:43:42.492086 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32106826-edda-4386-af34-106df0d8059c None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:42.498469 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:43:42.505058 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a7309b7-0ba4-4015-a152-b2f83df7a3d5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:42.507800 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a7309b7-0ba4-4015-a152-b2f83df7a3d5 None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:42.508294 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a7309b7-0ba4-4015-a152-b2f83df7a3d5 None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:42.508794 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a7309b7-0ba4-4015-a152-b2f83df7a3d5 None admin] PATH_INFO: `/v3/roles` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:42.509694 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a7309b7-0ba4-4015-a152-b2f83df7a3d5 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81578) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:43:42.510204 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a7309b7-0ba4-4015-a152-b2f83df7a3d5 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:42.511799 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a7309b7-0ba4-4015-a152-b2f83df7a3d5 None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:42.522878 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 375/755] 10.222.0.27 () {56 vars in 1099 bytes} [Tue Aug 30 01:43:42 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 225 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:43:42.528146 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a7309b7-0ba4-4015-a152-b2f83df7a3d5 None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:42.535655 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea18a0ae-420f-48e9-bc4f-22e3bfc48b83 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:42.536268 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea18a0ae-420f-48e9-bc4f-22e3bfc48b83 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:42.536549 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea18a0ae-420f-48e9-bc4f-22e3bfc48b83 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:42.537835 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea18a0ae-420f-48e9-bc4f-22e3bfc48b83 None admin] PATH_INFO: `/v3/projects` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:42.543162 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea18a0ae-420f-48e9-bc4f-22e3bfc48b83 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:43:42.543162 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea18a0ae-420f-48e9-bc4f-22e3bfc48b83 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:42.543162 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea18a0ae-420f-48e9-bc4f-22e3bfc48b83 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:42.553881 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 381/756] 10.222.0.27 () {56 vars in 1148 bytes} [Tue Aug 30 01:43:42 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 274 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:43:42.559722 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea18a0ae-420f-48e9-bc4f-22e3bfc48b83 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:42.686956 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea18a0ae-420f-48e9-bc4f-22e3bfc48b83 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:42.729038 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-664d7e55-2028-412d-9d50-a00ecf83fc59 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:42.729681 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-664d7e55-2028-412d-9d50-a00ecf83fc59 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:42.730555 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-664d7e55-2028-412d-9d50-a00ecf83fc59 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:42.730848 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-664d7e55-2028-412d-9d50-a00ecf83fc59 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:42.731579 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-664d7e55-2028-412d-9d50-a00ecf83fc59 None admin] RBAC: Adding query filter params (role.id=ead5fac86a0747ea8823a82efcea8ed7, scope.project.id=4562f3951fbe465280073e60a11e44e2, user.id=b4f0008fde424f22bc1f7307cf422774) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:43:42.731893 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-664d7e55-2028-412d-9d50-a00ecf83fc59 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:42.733327 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-664d7e55-2028-412d-9d50-a00ecf83fc59 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:42.741245 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 382/757] 10.222.0.27 () {56 vars in 1374 bytes} [Tue Aug 30 01:43:42 2022] GET /identity/v3/role_assignments?user.id=b4f0008fde424f22bc1f7307cf422774&scope.project.id=4562f3951fbe465280073e60a11e44e2&role.id=ead5fac86a0747ea8823a82efcea8ed7 => generated 259 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:43:42.765570 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a7309b7-0ba4-4015-a152-b2f83df7a3d5 None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:42.804145 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1630c28-00e9-455a-a2af-d5dbfe2b3506 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:42.804756 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1630c28-00e9-455a-a2af-d5dbfe2b3506 None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:42.805040 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1630c28-00e9-455a-a2af-d5dbfe2b3506 None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:42.805311 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1630c28-00e9-455a-a2af-d5dbfe2b3506 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:42.817499 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1630c28-00e9-455a-a2af-d5dbfe2b3506 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:42.819042 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1630c28-00e9-455a-a2af-d5dbfe2b3506 None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:42.837588 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application [None req-b1630c28-00e9-455a-a2af-d5dbfe2b3506 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 01:43:42.837588 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 01:43:42.837588 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 01:43:42.837588 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 01:43:42.837588 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 01:43:42.837588 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 01:43:42.837588 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 01:43:42.837588 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 01:43:42.837588 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 01:43:42.837588 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 01:43:42.837588 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 01:43:42.837588 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 01:43:42.837588 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 30 01:43:42.837588 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 30 01:43:42.837588 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 30 01:43:42.837588 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 30 01:43:42.838603 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 01:43:42.838603 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 01:43:42.838603 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 01:43:42.838603 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 01:43:42.838603 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 01:43:42.838603 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 01:43:42.838603 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 01:43:42.838603 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application with Lock( Aug 30 01:43:42.838603 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 01:43:42.838603 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._enter() Aug 30 01:43:42.838603 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 01:43:42.838603 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 01:43:42.838603 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 01:43:42.838603 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.creator() Aug 30 01:43:42.838603 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 01:43:42.838603 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application created_value = creator( Aug 30 01:43:42.838603 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 30 01:43:42.839511 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 30 01:43:42.839511 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 01:43:42.839511 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application Aug 30 01:43:42.839511 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 376/758] 10.222.0.27 () {56 vars in 1108 bytes} [Tue Aug 30 01:43:42 2022] GET /identity/v3/domains/Default => generated 87 bytes in 311 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 01:43:42.846214 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-664d7e55-2028-412d-9d50-a00ecf83fc59 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:42.886886 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:43:42.978914 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-664d7e55-2028-412d-9d50-a00ecf83fc59 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:43.033503 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcfa113d-4168-4951-9458-8f2500282cb9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:43.034126 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcfa113d-4168-4951-9458-8f2500282cb9 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:43.034406 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcfa113d-4168-4951-9458-8f2500282cb9 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:43.034677 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcfa113d-4168-4951-9458-8f2500282cb9 None admin] PATH_INFO: `/v3/domains` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:43.035331 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fcfa113d-4168-4951-9458-8f2500282cb9 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:43:43.035636 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fcfa113d-4168-4951-9458-8f2500282cb9 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:43.037065 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fcfa113d-4168-4951-9458-8f2500282cb9 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:43.055446 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 383/759] 10.222.0.27 () {56 vars in 1109 bytes} [Tue Aug 30 01:43:42 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 210 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:43:43.060936 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1630c28-00e9-455a-a2af-d5dbfe2b3506 None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:43.144009 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:43:43.213237 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1630c28-00e9-455a-a2af-d5dbfe2b3506 None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:43.247942 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0a12dc1-b644-43b5-a973-41fa5f3469f6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:43.248603 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0a12dc1-b644-43b5-a973-41fa5f3469f6 None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:43.248839 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0a12dc1-b644-43b5-a973-41fa5f3469f6 None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:43.249116 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0a12dc1-b644-43b5-a973-41fa5f3469f6 None admin] PATH_INFO: `/v3/users/nova` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:43.257114 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0a12dc1-b644-43b5-a973-41fa5f3469f6 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:43.258590 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0a12dc1-b644-43b5-a973-41fa5f3469f6 None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:43.269856 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application [None req-c0a12dc1-b644-43b5-a973-41fa5f3469f6 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Aug 30 01:43:43.269856 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 01:43:43.269856 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 01:43:43.269856 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 01:43:43.269856 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 01:43:43.269856 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 01:43:43.269856 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 01:43:43.269856 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 01:43:43.269856 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 01:43:43.269856 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 01:43:43.269856 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 01:43:43.269856 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 01:43:43.269856 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 30 01:43:43.269856 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 30 01:43:43.269856 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 30 01:43:43.269856 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 30 01:43:43.270777 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 01:43:43.270777 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 01:43:43.270777 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 30 01:43:43.270777 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 30 01:43:43.270777 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 30 01:43:43.270777 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 30 01:43:43.270777 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 01:43:43.270777 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 01:43:43.270777 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 01:43:43.270777 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 01:43:43.270777 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 01:43:43.270777 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application with Lock( Aug 30 01:43:43.270777 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 01:43:43.270777 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._enter() Aug 30 01:43:43.270777 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 01:43:43.270777 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 01:43:43.270777 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 01:43:43.270777 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.creator() Aug 30 01:43:43.272440 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 01:43:43.272440 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application created_value = creator( Aug 30 01:43:43.272440 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 30 01:43:43.272440 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 30 01:43:43.272440 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 30 01:43:43.272440 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 30 01:43:43.272440 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 30 01:43:43.272440 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 30 01:43:43.272440 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Aug 30 01:43:43.272440 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application Aug 30 01:43:43.272440 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 377/760] 10.222.0.27 () {56 vars in 1093 bytes} [Tue Aug 30 01:43:43 2022] GET /identity/v3/users/nova => generated 82 bytes in 211 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 01:43:43.282565 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcfa113d-4168-4951-9458-8f2500282cb9 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:43.433759 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:43:43.450697 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcfa113d-4168-4951-9458-8f2500282cb9 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:43.493577 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-062b899a-8354-42ca-993f-8e979b7a96e1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:43.494195 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-062b899a-8354-42ca-993f-8e979b7a96e1 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:43.494475 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-062b899a-8354-42ca-993f-8e979b7a96e1 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:43.494753 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-062b899a-8354-42ca-993f-8e979b7a96e1 None admin] PATH_INFO: `/v3/users` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:43.495491 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-062b899a-8354-42ca-993f-8e979b7a96e1 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:43:43.495795 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-062b899a-8354-42ca-993f-8e979b7a96e1 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:43.497219 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-062b899a-8354-42ca-993f-8e979b7a96e1 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:43.519843 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 384/761] 10.222.0.27 () {56 vars in 1133 bytes} [Tue Aug 30 01:43:43 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 238 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:43:43.525317 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0a12dc1-b644-43b5-a973-41fa5f3469f6 None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:43.549246 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28c32ccb-069e-43a7-b52c-66dc1b9e59ba None None] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:43.549880 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28c32ccb-069e-43a7-b52c-66dc1b9e59ba None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:43.550242 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28c32ccb-069e-43a7-b52c-66dc1b9e59ba None None] PATH_INFO: `/` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:43.552327 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 385/762] 10.222.0.27 () {54 vars in 927 bytes} [Tue Aug 30 01:43:43 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 01:43:43.557842 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0fc0082-76a8-4814-ac08-3a37d2a63bc1 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:43.558123 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0fc0082-76a8-4814-ac08-3a37d2a63bc1 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:43.558395 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0fc0082-76a8-4814-ac08-3a37d2a63bc1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:43.588056 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-d0fc0082-76a8-4814-ac08-3a37d2a63bc1 None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 01:43:43.589316 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-d0fc0082-76a8-4814-ac08-3a37d2a63bc1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:43:43.709943 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0a12dc1-b644-43b5-a973-41fa5f3469f6 None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:43.720253 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:43:43.731189 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 386/763] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Aug 30 01:43:43 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 174 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 30 01:43:43.750224 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44d281df-3243-4096-9eae-8c6eb28db683 None None] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:43.750598 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44d281df-3243-4096-9eae-8c6eb28db683 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:43.750805 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44d281df-3243-4096-9eae-8c6eb28db683 None None] PATH_INFO: `/` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:43.752642 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 387/764] 10.222.0.27 () {54 vars in 927 bytes} [Tue Aug 30 01:43:43 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 01:43:43.760111 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30610bf9-d6e0-44f6-8882-e9e7269a7a1e None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:43.760718 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30610bf9-d6e0-44f6-8882-e9e7269a7a1e None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:43.761201 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30610bf9-d6e0-44f6-8882-e9e7269a7a1e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:43.773006 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a535fe78-b1ff-411a-99c2-1fd6afe4f83d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:43.773969 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a535fe78-b1ff-411a-99c2-1fd6afe4f83d None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:43.779153 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a535fe78-b1ff-411a-99c2-1fd6afe4f83d None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:43.779665 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a535fe78-b1ff-411a-99c2-1fd6afe4f83d None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:43.792971 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a535fe78-b1ff-411a-99c2-1fd6afe4f83d None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:43.794473 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-30610bf9-d6e0-44f6-8882-e9e7269a7a1e None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 01:43:43.794941 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a535fe78-b1ff-411a-99c2-1fd6afe4f83d None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:43.795725 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-30610bf9-d6e0-44f6-8882-e9e7269a7a1e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:43:43.810252 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application [None req-a535fe78-b1ff-411a-99c2-1fd6afe4f83d None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 01:43:43.810252 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 01:43:43.810252 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 01:43:43.810252 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 01:43:43.810252 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 01:43:43.810252 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 01:43:43.810252 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 01:43:43.810252 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 01:43:43.810252 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 01:43:43.810252 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 01:43:43.810252 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 01:43:43.810252 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 01:43:43.810252 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 30 01:43:43.810252 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 30 01:43:43.810252 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 30 01:43:43.810252 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 30 01:43:43.811229 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 01:43:43.811229 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 01:43:43.811229 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 01:43:43.811229 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 01:43:43.811229 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 01:43:43.811229 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 01:43:43.811229 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 01:43:43.811229 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application with Lock( Aug 30 01:43:43.811229 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 01:43:43.811229 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._enter() Aug 30 01:43:43.811229 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 01:43:43.811229 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 01:43:43.811229 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 01:43:43.811229 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.creator() Aug 30 01:43:43.811229 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 01:43:43.811229 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application created_value = creator( Aug 30 01:43:43.811229 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 30 01:43:43.812130 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 30 01:43:43.812130 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 01:43:43.812130 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application Aug 30 01:43:43.814246 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 378/765] 10.222.0.27 () {56 vars in 1108 bytes} [Tue Aug 30 01:43:43 2022] GET /identity/v3/domains/Default => generated 87 bytes in 289 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 01:43:43.819780 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a535fe78-b1ff-411a-99c2-1fd6afe4f83d None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:43.946669 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 388/766] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Aug 30 01:43:43 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 186 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Aug 30 01:43:43.967340 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a535fe78-b1ff-411a-99c2-1fd6afe4f83d None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:44.003618 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b005e4e8-1a24-43df-b273-2abfc95b465c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:44.004595 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b005e4e8-1a24-43df-b273-2abfc95b465c None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:44.005080 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b005e4e8-1a24-43df-b273-2abfc95b465c None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:44.005777 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b005e4e8-1a24-43df-b273-2abfc95b465c None admin] PATH_INFO: `/v3/domains` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:44.006898 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b005e4e8-1a24-43df-b273-2abfc95b465c None admin] RBAC: Adding query filter params (name=Default) {{(pid=81578) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:43:44.007413 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b005e4e8-1a24-43df-b273-2abfc95b465c None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:44.008986 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b005e4e8-1a24-43df-b273-2abfc95b465c None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:44.022847 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 379/767] 10.222.0.27 () {56 vars in 1109 bytes} [Tue Aug 30 01:43:43 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 203 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:43:44.030542 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30610bf9-d6e0-44f6-8882-e9e7269a7a1e None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:44.033321 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:43:44.104205 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b005e4e8-1a24-43df-b273-2abfc95b465c None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:44.136676 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:43:44.169561 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30610bf9-d6e0-44f6-8882-e9e7269a7a1e None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:44.206485 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb152112-522f-4983-81f9-c2af9d4f3b90 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:44.207485 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb152112-522f-4983-81f9-c2af9d4f3b90 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:44.207972 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb152112-522f-4983-81f9-c2af9d4f3b90 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:44.208481 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb152112-522f-4983-81f9-c2af9d4f3b90 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:44.216731 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb152112-522f-4983-81f9-c2af9d4f3b90 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:44.218562 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb152112-522f-4983-81f9-c2af9d4f3b90 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:44.231263 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application [None req-cb152112-522f-4983-81f9-c2af9d4f3b90 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 30 01:43:44.231263 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 01:43:44.231263 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 01:43:44.231263 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 01:43:44.231263 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 01:43:44.231263 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 01:43:44.231263 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 01:43:44.231263 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 01:43:44.231263 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 01:43:44.231263 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 01:43:44.231263 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 01:43:44.231263 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 01:43:44.231263 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 30 01:43:44.231263 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 30 01:43:44.231263 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 30 01:43:44.232259 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 30 01:43:44.232259 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 01:43:44.232259 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 01:43:44.232259 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 01:43:44.232259 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 01:43:44.232259 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 01:43:44.232259 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 01:43:44.232259 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 01:43:44.232259 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application with Lock( Aug 30 01:43:44.232259 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 01:43:44.232259 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._enter() Aug 30 01:43:44.232259 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 01:43:44.232259 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 01:43:44.232259 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 01:43:44.232259 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.creator() Aug 30 01:43:44.232259 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 01:43:44.232259 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application created_value = creator( Aug 30 01:43:44.233242 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 30 01:43:44.233242 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 30 01:43:44.233242 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 30 01:43:44.233242 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 30 01:43:44.233242 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 30 01:43:44.233242 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 30 01:43:44.233242 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 30 01:43:44.233242 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Aug 30 01:43:44.235873 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 389/768] 10.222.0.27 () {56 vars in 1111 bytes} [Tue Aug 30 01:43:44 2022] GET /identity/v3/projects/service => generated 88 bytes in 206 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 01:43:44.242146 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb152112-522f-4983-81f9-c2af9d4f3b90 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:44.328289 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b005e4e8-1a24-43df-b273-2abfc95b465c None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:44.340100 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:43:44.368917 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aade45f4-e61c-4d90-a105-57ff557a4147 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:44.369819 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aade45f4-e61c-4d90-a105-57ff557a4147 None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:44.370366 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aade45f4-e61c-4d90-a105-57ff557a4147 None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:44.370795 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aade45f4-e61c-4d90-a105-57ff557a4147 None admin] PATH_INFO: `/v3/services/block-storage` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:44.371800 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aade45f4-e61c-4d90-a105-57ff557a4147 None admin] RBAC: Authorizing `identity:get_service(service_id=block-storage)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:44.373327 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aade45f4-e61c-4d90-a105-57ff557a4147 None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:44.386863 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application [None req-aade45f4-e61c-4d90-a105-57ff557a4147 None admin] Could not find service: block-storage.: keystone.exception.ServiceNotFound: Could not find service: block-storage. Aug 30 01:43:44.386863 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 01:43:44.386863 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Aug 30 01:43:44.386863 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Aug 30 01:43:44.386863 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Aug 30 01:43:44.386863 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Aug 30 01:43:44.386863 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Aug 30 01:43:44.386863 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 30 01:43:44.386863 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Aug 30 01:43:44.386863 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application Aug 30 01:43:44.386863 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 30 01:43:44.386863 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application Aug 30 01:43:44.386863 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 01:43:44.386863 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 01:43:44.386863 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 01:43:44.386863 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 01:43:44.386863 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 01:43:44.388077 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 01:43:44.388077 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 01:43:44.388077 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 01:43:44.388077 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 01:43:44.388077 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 01:43:44.388077 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 01:43:44.388077 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Aug 30 01:43:44.388077 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._get_service(service_id) Aug 30 01:43:44.388077 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Aug 30 01:43:44.388077 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Aug 30 01:43:44.388077 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 01:43:44.388077 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 01:43:44.388077 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 01:43:44.388077 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 01:43:44.388077 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 01:43:44.388077 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 01:43:44.389093 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 01:43:44.389093 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application with Lock( Aug 30 01:43:44.389093 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 01:43:44.389093 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._enter() Aug 30 01:43:44.389093 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 01:43:44.389093 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 01:43:44.389093 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 01:43:44.389093 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.creator() Aug 30 01:43:44.389093 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 01:43:44.389093 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application created_value = creator( Aug 30 01:43:44.389093 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Aug 30 01:43:44.389093 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 30 01:43:44.389093 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Aug 30 01:43:44.389093 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application Aug 30 01:43:44.390815 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 380/769] 10.222.0.27 () {56 vars in 1129 bytes} [Tue Aug 30 01:43:44 2022] GET /identity/v3/services/block-storage => generated 94 bytes in 287 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 30 01:43:44.395647 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aade45f4-e61c-4d90-a105-57ff557a4147 None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:44.397836 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb152112-522f-4983-81f9-c2af9d4f3b90 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:44.442153 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d31a5d0c-efd7-429c-863f-8715106ea64e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:44.442761 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d31a5d0c-efd7-429c-863f-8715106ea64e None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:44.443068 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d31a5d0c-efd7-429c-863f-8715106ea64e None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:44.443495 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d31a5d0c-efd7-429c-863f-8715106ea64e None admin] PATH_INFO: `/v3/projects` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:44.444307 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d31a5d0c-efd7-429c-863f-8715106ea64e None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:43:44.444609 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d31a5d0c-efd7-429c-863f-8715106ea64e None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:44.446577 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d31a5d0c-efd7-429c-863f-8715106ea64e None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:44.467025 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 390/770] 10.222.0.27 () {56 vars in 1148 bytes} [Tue Aug 30 01:43:44 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 221 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 30 01:43:44.468304 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d31a5d0c-efd7-429c-863f-8715106ea64e None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:44.580818 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d31a5d0c-efd7-429c-863f-8715106ea64e None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:44.618585 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:43:44.642990 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aade45f4-e61c-4d90-a105-57ff557a4147 None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:44.682861 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62716952-3789-4e4a-9902-3aebbf07f1ee None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:44.683512 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62716952-3789-4e4a-9902-3aebbf07f1ee None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:44.683790 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62716952-3789-4e4a-9902-3aebbf07f1ee None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:44.684059 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62716952-3789-4e4a-9902-3aebbf07f1ee None admin] PATH_INFO: `/v3/services/network` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:44.684653 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62716952-3789-4e4a-9902-3aebbf07f1ee None admin] RBAC: Authorizing `identity:get_service(service_id=network)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:44.686145 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62716952-3789-4e4a-9902-3aebbf07f1ee None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:44.696653 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application [None req-62716952-3789-4e4a-9902-3aebbf07f1ee None admin] Could not find service: network.: keystone.exception.ServiceNotFound: Could not find service: network. Aug 30 01:43:44.696653 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 01:43:44.696653 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Aug 30 01:43:44.696653 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Aug 30 01:43:44.696653 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Aug 30 01:43:44.696653 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Aug 30 01:43:44.696653 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Aug 30 01:43:44.696653 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 30 01:43:44.696653 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Aug 30 01:43:44.696653 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application Aug 30 01:43:44.696653 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 30 01:43:44.696653 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application Aug 30 01:43:44.696653 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 01:43:44.696653 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 01:43:44.696653 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 01:43:44.696653 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 01:43:44.696653 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 01:43:44.697751 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 01:43:44.697751 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 01:43:44.697751 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 01:43:44.697751 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 01:43:44.697751 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 01:43:44.697751 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 01:43:44.697751 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Aug 30 01:43:44.697751 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._get_service(service_id) Aug 30 01:43:44.697751 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Aug 30 01:43:44.697751 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Aug 30 01:43:44.697751 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 01:43:44.697751 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 01:43:44.697751 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 01:43:44.697751 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 01:43:44.697751 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 01:43:44.697751 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 01:43:44.702639 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 01:43:44.702639 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application with Lock( Aug 30 01:43:44.702639 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 01:43:44.702639 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._enter() Aug 30 01:43:44.702639 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 01:43:44.702639 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 01:43:44.702639 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 01:43:44.702639 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.creator() Aug 30 01:43:44.702639 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 01:43:44.702639 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application created_value = creator( Aug 30 01:43:44.702639 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Aug 30 01:43:44.702639 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 30 01:43:44.702639 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Aug 30 01:43:44.702639 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application Aug 30 01:43:44.702639 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 381/771] 10.222.0.27 () {56 vars in 1111 bytes} [Tue Aug 30 01:43:44 2022] GET /identity/v3/services/network => generated 88 bytes in 307 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 01:43:44.703438 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62716952-3789-4e4a-9902-3aebbf07f1ee None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:44.737177 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:43:44.750219 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f21e958-2d49-4fe1-92d7-12a09abd6fff None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:44.751095 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f21e958-2d49-4fe1-92d7-12a09abd6fff None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:44.751517 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f21e958-2d49-4fe1-92d7-12a09abd6fff None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:44.752081 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f21e958-2d49-4fe1-92d7-12a09abd6fff None admin] PATH_INFO: `/v3/services` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:44.752727 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f21e958-2d49-4fe1-92d7-12a09abd6fff None admin] RBAC: Adding query filter params (name=block-storage) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:43:44.753028 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f21e958-2d49-4fe1-92d7-12a09abd6fff None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:44.754551 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f21e958-2d49-4fe1-92d7-12a09abd6fff None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:44.761108 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 391/772] 10.222.0.27 () {56 vars in 1124 bytes} [Tue Aug 30 01:43:44 2022] GET /identity/v3/services?name=block-storage => generated 130 bytes in 293 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:43:44.762422 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5bfa544-f371-4987-8dc6-f9d5eda1e679 None None] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:44.762703 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5bfa544-f371-4987-8dc6-f9d5eda1e679 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:44.762973 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5bfa544-f371-4987-8dc6-f9d5eda1e679 None None] PATH_INFO: `/` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:44.764292 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 392/773] 10.222.0.27 () {54 vars in 927 bytes} [Tue Aug 30 01:43:44 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 01:43:44.768845 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5bfa544-f371-4987-8dc6-f9d5eda1e679 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:44.819677 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62716952-3789-4e4a-9902-3aebbf07f1ee None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:44.858105 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-157d44ca-abfa-46ef-9aab-4f9dc8af22b8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:44.858681 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-157d44ca-abfa-46ef-9aab-4f9dc8af22b8 None admin] REQUEST_METHOD: `PUT` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:44.858969 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-157d44ca-abfa-46ef-9aab-4f9dc8af22b8 None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:44.859226 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-157d44ca-abfa-46ef-9aab-4f9dc8af22b8 None admin] PATH_INFO: `/v3/projects/4562f3951fbe465280073e60a11e44e2/users/d64f2e71361644589bec8a06e6eceed1/roles/ead5fac86a0747ea8823a82efcea8ed7` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:44.861110 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-157d44ca-abfa-46ef-9aab-4f9dc8af22b8 None admin] RBAC: Authorizing `identity:create_grant(project_id=4562f3951fbe465280073e60a11e44e2, user_id=d64f2e71361644589bec8a06e6eceed1, role_id=ead5fac86a0747ea8823a82efcea8ed7)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:44.862548 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-157d44ca-abfa-46ef-9aab-4f9dc8af22b8 None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:44.879873 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5bfa544-f371-4987-8dc6-f9d5eda1e679 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:44.913946 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00df5fd3-c00a-45ed-9f81-1528b8fdac78 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:44.914541 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00df5fd3-c00a-45ed-9f81-1528b8fdac78 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:44.914818 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00df5fd3-c00a-45ed-9f81-1528b8fdac78 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:44.915086 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00df5fd3-c00a-45ed-9f81-1528b8fdac78 None admin] PATH_INFO: `/v3/services` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:44.915713 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00df5fd3-c00a-45ed-9f81-1528b8fdac78 None admin] RBAC: Adding query filter params (name=network) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:43:44.916012 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00df5fd3-c00a-45ed-9f81-1528b8fdac78 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:44.917365 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00df5fd3-c00a-45ed-9f81-1528b8fdac78 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:44.921711 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 393/774] 10.222.0.27 () {56 vars in 1112 bytes} [Tue Aug 30 01:43:44 2022] GET /identity/v3/services?name=network => generated 124 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:43:44.922473 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00df5fd3-c00a-45ed-9f81-1528b8fdac78 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:44.947188 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 382/775] 10.222.0.27 () {58 vars in 1439 bytes} [Tue Aug 30 01:43:44 2022] PUT /identity/v3/projects/4562f3951fbe465280073e60a11e44e2/users/d64f2e71361644589bec8a06e6eceed1/roles/ead5fac86a0747ea8823a82efcea8ed7 => generated 0 bytes in 248 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 01:43:44.948685 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-207e9391-c4e9-43e4-a9c1-59f1cb733e17 None None] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:44.951971 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-207e9391-c4e9-43e4-a9c1-59f1cb733e17 None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:44.951971 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-207e9391-c4e9-43e4-a9c1-59f1cb733e17 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:44.956980 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:43:44.969830 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.auth.core [None req-207e9391-c4e9-43e4-a9c1-59f1cb733e17 None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81578) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 01:43:44.971351 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-207e9391-c4e9-43e4-a9c1-59f1cb733e17 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:43:44.989060 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:43:45.065517 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00df5fd3-c00a-45ed-9f81-1528b8fdac78 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:45.066174 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 383/776] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Aug 30 01:43:44 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 118 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 30 01:43:45.068631 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-207e9391-c4e9-43e4-a9c1-59f1cb733e17 None None] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:45.108406 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2b2722b-8040-4ac2-b18a-ed0b18f0aaf1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:45.109180 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2b2722b-8040-4ac2-b18a-ed0b18f0aaf1 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:45.112284 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2b2722b-8040-4ac2-b18a-ed0b18f0aaf1 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:45.112284 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2b2722b-8040-4ac2-b18a-ed0b18f0aaf1 None admin] PATH_INFO: `/v3/services` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:45.112284 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2b2722b-8040-4ac2-b18a-ed0b18f0aaf1 None admin] RBAC: Adding query filter params (type=block-storage) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:43:45.112284 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2b2722b-8040-4ac2-b18a-ed0b18f0aaf1 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:45.112845 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2b2722b-8040-4ac2-b18a-ed0b18f0aaf1 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:45.119427 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 394/777] 10.222.0.27 () {56 vars in 1124 bytes} [Tue Aug 30 01:43:44 2022] GET /identity/v3/services?type=block-storage => generated 367 bytes in 197 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:43:45.125128 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2b2722b-8040-4ac2-b18a-ed0b18f0aaf1 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:45.212843 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-207e9391-c4e9-43e4-a9c1-59f1cb733e17 None None] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:45.240292 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2b2722b-8040-4ac2-b18a-ed0b18f0aaf1 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:45.250066 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9763bb37-1423-462e-bc7c-4694b76da0c8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:45.250610 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9763bb37-1423-462e-bc7c-4694b76da0c8 None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:45.250888 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9763bb37-1423-462e-bc7c-4694b76da0c8 None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:45.251159 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9763bb37-1423-462e-bc7c-4694b76da0c8 None admin] PATH_INFO: `/v3/services` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:45.251798 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9763bb37-1423-462e-bc7c-4694b76da0c8 None admin] RBAC: Adding query filter params (type=network) {{(pid=81578) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:43:45.252099 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9763bb37-1423-462e-bc7c-4694b76da0c8 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:45.253473 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9763bb37-1423-462e-bc7c-4694b76da0c8 None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:45.254967 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:43:45.257181 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:43:45.260793 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 384/778] 10.222.0.27 () {56 vars in 1112 bytes} [Tue Aug 30 01:43:45 2022] GET /identity/v3/services?type=network => generated 350 bytes in 194 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:43:45.266624 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5eb6145b-bd58-434e-8b1a-63d064f03e95 None None] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:45.267141 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5eb6145b-bd58-434e-8b1a-63d064f03e95 None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:45.267583 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5eb6145b-bd58-434e-8b1a-63d064f03e95 None None] PATH_INFO: `/` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:45.277869 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 385/779] 10.222.0.27 () {54 vars in 927 bytes} [Tue Aug 30 01:43:45 2022] GET /identity => generated 270 bytes in 8 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 01:43:45.277869 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5eb6145b-bd58-434e-8b1a-63d064f03e95 None None] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:45.279211 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6bff539-f930-4927-8ed9-68ad0cb063ea None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:45.279824 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6bff539-f930-4927-8ed9-68ad0cb063ea None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:45.280159 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6bff539-f930-4927-8ed9-68ad0cb063ea None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:45.280474 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6bff539-f930-4927-8ed9-68ad0cb063ea None admin] PATH_INFO: `/v3/endpoints` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:45.281178 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6bff539-f930-4927-8ed9-68ad0cb063ea None admin] RBAC: Adding query filter params (interface=public, service_id=8100d9b5be9d463d8f66affc3f5058f0, region_id=RegionOne) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:43:45.281499 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6bff539-f930-4927-8ed9-68ad0cb063ea None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:45.283447 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6bff539-f930-4927-8ed9-68ad0cb063ea None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:45.294852 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 395/780] 10.222.0.27 () {56 vars in 1251 bytes} [Tue Aug 30 01:43:45 2022] GET /identity/v3/endpoints?service_id=8100d9b5be9d463d8f66affc3f5058f0&interface=public®ion_id=RegionOne => generated 194 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 30 01:43:45.296096 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44ff668e-96fc-4ba4-a48e-9f912de60628 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:45.296375 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44ff668e-96fc-4ba4-a48e-9f912de60628 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:45.296733 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44ff668e-96fc-4ba4-a48e-9f912de60628 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:45.326123 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-44ff668e-96fc-4ba4-a48e-9f912de60628 None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 01:43:45.327491 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-44ff668e-96fc-4ba4-a48e-9f912de60628 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:43:45.407748 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5eb6145b-bd58-434e-8b1a-63d064f03e95 None None] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:45.408586 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 396/781] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Aug 30 01:43:45 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 113 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 30 01:43:45.409433 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44ff668e-96fc-4ba4-a48e-9f912de60628 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:45.459829 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43e1b668-dfb6-4025-99f5-c474c9374ec5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:45.461091 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43e1b668-dfb6-4025-99f5-c474c9374ec5 None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:45.462016 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43e1b668-dfb6-4025-99f5-c474c9374ec5 None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:45.462501 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43e1b668-dfb6-4025-99f5-c474c9374ec5 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:45.463413 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43e1b668-dfb6-4025-99f5-c474c9374ec5 None admin] RBAC: Adding query filter params (interface=public, service_id=e340008443044b8f96ba46f48ddd28f1, region_id=RegionOne) {{(pid=81578) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:43:45.463929 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43e1b668-dfb6-4025-99f5-c474c9374ec5 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:45.466342 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43e1b668-dfb6-4025-99f5-c474c9374ec5 None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:45.482354 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 386/782] 10.222.0.27 () {56 vars in 1251 bytes} [Tue Aug 30 01:43:45 2022] GET /identity/v3/endpoints?service_id=e340008443044b8f96ba46f48ddd28f1&interface=public®ion_id=RegionOne => generated 194 bytes in 208 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:43:45.482354 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43e1b668-dfb6-4025-99f5-c474c9374ec5 None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:45.533972 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44ff668e-96fc-4ba4-a48e-9f912de60628 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:45.546154 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:43:45.567440 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:43:45.572750 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fe233cb-25ed-49c0-b6ca-82073c49116c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:45.573617 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fe233cb-25ed-49c0-b6ca-82073c49116c None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:45.574035 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fe233cb-25ed-49c0-b6ca-82073c49116c None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:45.574456 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fe233cb-25ed-49c0-b6ca-82073c49116c None admin] PATH_INFO: `/v3/services` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:45.575196 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fe233cb-25ed-49c0-b6ca-82073c49116c None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:45.576697 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fe233cb-25ed-49c0-b6ca-82073c49116c None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:45.584679 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 397/783] 10.222.0.27 () {56 vars in 1087 bytes} [Tue Aug 30 01:43:45 2022] GET /identity/v3/services => generated 772 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:43:45.585777 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fe233cb-25ed-49c0-b6ca-82073c49116c None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:45.622436 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43e1b668-dfb6-4025-99f5-c474c9374ec5 None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:45.677710 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82d709ca-768c-4813-a138-d4c458eacbd0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:45.684566 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82d709ca-768c-4813-a138-d4c458eacbd0 None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:45.685208 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82d709ca-768c-4813-a138-d4c458eacbd0 None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:45.688211 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82d709ca-768c-4813-a138-d4c458eacbd0 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:45.701782 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82d709ca-768c-4813-a138-d4c458eacbd0 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 30 01:43:45.707272 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application [None req-82d709ca-768c-4813-a138-d4c458eacbd0 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Aug 30 01:43:45.707272 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 01:43:45.707272 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 01:43:45.707272 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 01:43:45.707272 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 01:43:45.707272 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 01:43:45.707272 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 01:43:45.707272 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 01:43:45.707272 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 01:43:45.707272 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 01:43:45.707272 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 01:43:45.707272 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 01:43:45.707272 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 30 01:43:45.707272 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 30 01:43:45.707272 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 30 01:43:45.707272 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 30 01:43:45.708285 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 30 01:43:45.708285 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 30 01:43:45.708285 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 30 01:43:45.708285 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 30 01:43:45.708285 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 01:43:45.708285 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 01:43:45.708285 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 01:43:45.708285 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 01:43:45.708285 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 01:43:45.708285 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 01:43:45.708285 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 01:43:45.708285 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application with Lock( Aug 30 01:43:45.708285 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 01:43:45.708285 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._enter() Aug 30 01:43:45.708285 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 01:43:45.708285 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 01:43:45.710368 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 01:43:45.710368 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.creator() Aug 30 01:43:45.710368 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 01:43:45.710368 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application created_value = creator( Aug 30 01:43:45.710368 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 30 01:43:45.710368 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 30 01:43:45.710368 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 30 01:43:45.710368 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 30 01:43:45.710368 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 30 01:43:45.710368 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 30 01:43:45.710368 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Aug 30 01:43:45.710368 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application Aug 30 01:43:45.714050 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 387/784] 10.222.0.27 () {56 vars in 1102 bytes} [Tue Aug 30 01:43:45 2022] GET /identity/v3/roles/service => generated 85 bytes in 235 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 01:43:45.715269 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82d709ca-768c-4813-a138-d4c458eacbd0 None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:45.730829 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fe233cb-25ed-49c0-b6ca-82073c49116c None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:45.785475 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc0b9741-47d1-4e45-966e-6e14bc7f584e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:45.786099 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc0b9741-47d1-4e45-966e-6e14bc7f584e None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:45.786375 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc0b9741-47d1-4e45-966e-6e14bc7f584e None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:45.787168 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc0b9741-47d1-4e45-966e-6e14bc7f584e None admin] PATH_INFO: `/v3/services` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:45.787780 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc0b9741-47d1-4e45-966e-6e14bc7f584e None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:45.789138 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc0b9741-47d1-4e45-966e-6e14bc7f584e None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:45.795916 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 398/785] 10.222.0.27 () {56 vars in 1087 bytes} [Tue Aug 30 01:43:45 2022] GET /identity/v3/services => generated 772 bytes in 210 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:43:45.796886 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc0b9741-47d1-4e45-966e-6e14bc7f584e None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:45.841975 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82d709ca-768c-4813-a138-d4c458eacbd0 None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:45.843893 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:43:45.870432 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:43:45.886044 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fd7a364-a43f-4f1d-94c7-bcd1a4570d81 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:45.887078 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fd7a364-a43f-4f1d-94c7-bcd1a4570d81 None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:45.887565 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fd7a364-a43f-4f1d-94c7-bcd1a4570d81 None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:45.888041 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fd7a364-a43f-4f1d-94c7-bcd1a4570d81 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:45.901891 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fd7a364-a43f-4f1d-94c7-bcd1a4570d81 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 30 01:43:45.907557 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application [None req-0fd7a364-a43f-4f1d-94c7-bcd1a4570d81 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 30 01:43:45.907557 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 01:43:45.907557 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 01:43:45.907557 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 01:43:45.907557 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 01:43:45.907557 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 01:43:45.907557 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 01:43:45.907557 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 01:43:45.907557 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 01:43:45.907557 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 01:43:45.907557 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 01:43:45.907557 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 01:43:45.907557 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 30 01:43:45.907557 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 30 01:43:45.907557 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 30 01:43:45.907557 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 30 01:43:45.908587 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 30 01:43:45.908587 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 30 01:43:45.908587 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 30 01:43:45.908587 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 30 01:43:45.908587 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 01:43:45.908587 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 01:43:45.908587 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 01:43:45.908587 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 01:43:45.908587 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 01:43:45.908587 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 01:43:45.908587 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 01:43:45.908587 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application with Lock( Aug 30 01:43:45.908587 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 01:43:45.908587 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._enter() Aug 30 01:43:45.908587 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 01:43:45.908587 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 01:43:45.909434 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 01:43:45.909434 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.creator() Aug 30 01:43:45.909434 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 01:43:45.909434 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application created_value = creator( Aug 30 01:43:45.909434 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 30 01:43:45.909434 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 30 01:43:45.909434 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 30 01:43:45.909434 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 30 01:43:45.909434 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 30 01:43:45.909434 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 30 01:43:45.909434 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 30 01:43:45.909434 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application Aug 30 01:43:45.917574 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 388/786] 10.222.0.27 () {56 vars in 1096 bytes} [Tue Aug 30 01:43:45 2022] GET /identity/v3/roles/admin => generated 83 bytes in 198 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 01:43:45.919315 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc0b9741-47d1-4e45-966e-6e14bc7f584e None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:45.919968 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fd7a364-a43f-4f1d-94c7-bcd1a4570d81 None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:45.960068 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40a40cec-bfb5-49d1-98c2-ddd9f141149c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:45.960781 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40a40cec-bfb5-49d1-98c2-ddd9f141149c None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:45.961152 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40a40cec-bfb5-49d1-98c2-ddd9f141149c None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:45.961539 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40a40cec-bfb5-49d1-98c2-ddd9f141149c None admin] PATH_INFO: `/v3/roles` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:45.962369 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40a40cec-bfb5-49d1-98c2-ddd9f141149c None admin] RBAC: Adding query filter params (name=service) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:43:45.962769 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40a40cec-bfb5-49d1-98c2-ddd9f141149c None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:45.964494 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40a40cec-bfb5-49d1-98c2-ddd9f141149c None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:45.974698 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 399/787] 10.222.0.27 () {56 vars in 1103 bytes} [Tue Aug 30 01:43:45 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:43:45.982152 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40a40cec-bfb5-49d1-98c2-ddd9f141149c None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:46.070961 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fd7a364-a43f-4f1d-94c7-bcd1a4570d81 None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:46.105307 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:43:46.109489 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40a40cec-bfb5-49d1-98c2-ddd9f141149c None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:46.110085 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d862437e-3c63-4396-87c9-ac985b4a9be1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:46.110698 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d862437e-3c63-4396-87c9-ac985b4a9be1 None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:46.110974 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d862437e-3c63-4396-87c9-ac985b4a9be1 None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:46.111241 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d862437e-3c63-4396-87c9-ac985b4a9be1 None admin] PATH_INFO: `/v3/roles` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:46.112078 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d862437e-3c63-4396-87c9-ac985b4a9be1 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81578) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:43:46.112380 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d862437e-3c63-4396-87c9-ac985b4a9be1 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:46.116080 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d862437e-3c63-4396-87c9-ac985b4a9be1 None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:46.132225 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 389/788] 10.222.0.27 () {56 vars in 1099 bytes} [Tue Aug 30 01:43:45 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 208 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:43:46.134834 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d862437e-3c63-4396-87c9-ac985b4a9be1 None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:46.157401 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6866cbe-bcbf-4fd1-8ddd-7f0276c2233f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:46.163153 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6866cbe-bcbf-4fd1-8ddd-7f0276c2233f None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:46.163153 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6866cbe-bcbf-4fd1-8ddd-7f0276c2233f None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:46.163153 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6866cbe-bcbf-4fd1-8ddd-7f0276c2233f None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:46.169862 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6866cbe-bcbf-4fd1-8ddd-7f0276c2233f None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:46.171325 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6866cbe-bcbf-4fd1-8ddd-7f0276c2233f None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:46.188575 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application [None req-d6866cbe-bcbf-4fd1-8ddd-7f0276c2233f None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 01:43:46.188575 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 01:43:46.188575 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 01:43:46.188575 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 01:43:46.188575 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 01:43:46.188575 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 01:43:46.188575 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 01:43:46.188575 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 01:43:46.188575 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 01:43:46.188575 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 01:43:46.188575 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 01:43:46.188575 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 01:43:46.188575 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 30 01:43:46.188575 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 30 01:43:46.188575 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 30 01:43:46.188575 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 30 01:43:46.189823 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 01:43:46.189823 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 01:43:46.189823 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 01:43:46.189823 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 01:43:46.189823 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 01:43:46.189823 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 01:43:46.189823 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 01:43:46.189823 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application with Lock( Aug 30 01:43:46.189823 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 01:43:46.189823 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._enter() Aug 30 01:43:46.189823 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 01:43:46.189823 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 01:43:46.189823 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 01:43:46.189823 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.creator() Aug 30 01:43:46.189823 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 01:43:46.189823 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application created_value = creator( Aug 30 01:43:46.189823 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 30 01:43:46.193804 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 30 01:43:46.193804 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 01:43:46.193804 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Aug 30 01:43:46.193804 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 400/789] 10.222.0.27 () {56 vars in 1108 bytes} [Tue Aug 30 01:43:45 2022] GET /identity/v3/domains/Default => generated 87 bytes in 210 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 30 01:43:46.195906 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6866cbe-bcbf-4fd1-8ddd-7f0276c2233f None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:46.221894 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:43:46.292515 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d862437e-3c63-4396-87c9-ac985b4a9be1 None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:46.328107 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e458e1a1-d08d-4ff3-85c7-5ca143da4e5d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:46.328748 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e458e1a1-d08d-4ff3-85c7-5ca143da4e5d None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:46.329025 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e458e1a1-d08d-4ff3-85c7-5ca143da4e5d None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:46.329295 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e458e1a1-d08d-4ff3-85c7-5ca143da4e5d None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:46.341033 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e458e1a1-d08d-4ff3-85c7-5ca143da4e5d None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:46.342716 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e458e1a1-d08d-4ff3-85c7-5ca143da4e5d None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:46.360265 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application [None req-e458e1a1-d08d-4ff3-85c7-5ca143da4e5d None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 01:43:46.360265 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 01:43:46.360265 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 01:43:46.360265 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 01:43:46.360265 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 01:43:46.360265 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 01:43:46.360265 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 01:43:46.360265 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 01:43:46.360265 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 01:43:46.360265 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 01:43:46.360265 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 01:43:46.360265 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 01:43:46.360265 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 30 01:43:46.360265 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 30 01:43:46.360265 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 30 01:43:46.360265 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 30 01:43:46.361404 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 01:43:46.361404 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 01:43:46.361404 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 01:43:46.361404 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 01:43:46.361404 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 01:43:46.361404 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 01:43:46.361404 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 01:43:46.361404 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application with Lock( Aug 30 01:43:46.361404 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 01:43:46.361404 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._enter() Aug 30 01:43:46.361404 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 01:43:46.361404 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 01:43:46.361404 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 01:43:46.361404 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.creator() Aug 30 01:43:46.361404 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 01:43:46.361404 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application created_value = creator( Aug 30 01:43:46.361404 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 30 01:43:46.362323 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 30 01:43:46.362323 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 01:43:46.362323 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application Aug 30 01:43:46.362323 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 390/790] 10.222.0.27 () {56 vars in 1108 bytes} [Tue Aug 30 01:43:46 2022] GET /identity/v3/domains/Default => generated 87 bytes in 227 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 01:43:46.370955 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e458e1a1-d08d-4ff3-85c7-5ca143da4e5d None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:46.401991 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6866cbe-bcbf-4fd1-8ddd-7f0276c2233f None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:46.475986 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50904da3-0197-4c49-8480-d4d1c4e3b964 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:46.476849 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50904da3-0197-4c49-8480-d4d1c4e3b964 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:46.477038 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50904da3-0197-4c49-8480-d4d1c4e3b964 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:46.477313 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50904da3-0197-4c49-8480-d4d1c4e3b964 None admin] PATH_INFO: `/v3/domains` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:46.477999 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50904da3-0197-4c49-8480-d4d1c4e3b964 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:43:46.478305 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50904da3-0197-4c49-8480-d4d1c4e3b964 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:46.479688 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50904da3-0197-4c49-8480-d4d1c4e3b964 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:46.499829 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 401/791] 10.222.0.27 () {56 vars in 1109 bytes} [Tue Aug 30 01:43:46 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 304 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 30 01:43:46.504780 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50904da3-0197-4c49-8480-d4d1c4e3b964 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:46.527932 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:43:46.538966 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e458e1a1-d08d-4ff3-85c7-5ca143da4e5d None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:46.551018 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:43:46.576567 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c083e5b3-0117-4f2c-b4a5-332e1ddb7145 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:46.582572 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c083e5b3-0117-4f2c-b4a5-332e1ddb7145 None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:46.582572 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c083e5b3-0117-4f2c-b4a5-332e1ddb7145 None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:46.582572 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c083e5b3-0117-4f2c-b4a5-332e1ddb7145 None admin] PATH_INFO: `/v3/domains` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:46.582572 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c083e5b3-0117-4f2c-b4a5-332e1ddb7145 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81578) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:43:46.582572 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c083e5b3-0117-4f2c-b4a5-332e1ddb7145 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:46.582572 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c083e5b3-0117-4f2c-b4a5-332e1ddb7145 None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:46.594572 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 391/792] 10.222.0.27 () {56 vars in 1109 bytes} [Tue Aug 30 01:43:46 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 224 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:43:46.610876 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c083e5b3-0117-4f2c-b4a5-332e1ddb7145 None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:46.665252 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50904da3-0197-4c49-8480-d4d1c4e3b964 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:46.708804 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8366a98c-f98f-4a4b-b916-ac3275d9318e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:46.710601 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8366a98c-f98f-4a4b-b916-ac3275d9318e None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:46.711028 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8366a98c-f98f-4a4b-b916-ac3275d9318e None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:46.715991 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8366a98c-f98f-4a4b-b916-ac3275d9318e None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:46.728610 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8366a98c-f98f-4a4b-b916-ac3275d9318e None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:46.729979 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8366a98c-f98f-4a4b-b916-ac3275d9318e None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:46.750726 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application [None req-8366a98c-f98f-4a4b-b916-ac3275d9318e None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Aug 30 01:43:46.750726 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 01:43:46.750726 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 01:43:46.750726 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 01:43:46.750726 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 01:43:46.750726 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 01:43:46.750726 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 01:43:46.750726 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 01:43:46.750726 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 01:43:46.750726 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 01:43:46.750726 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 01:43:46.750726 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 01:43:46.750726 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 30 01:43:46.750726 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 30 01:43:46.750726 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 30 01:43:46.750726 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 30 01:43:46.751718 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 01:43:46.751718 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 01:43:46.751718 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 30 01:43:46.751718 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 30 01:43:46.751718 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 30 01:43:46.751718 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 30 01:43:46.751718 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 01:43:46.751718 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 01:43:46.751718 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 01:43:46.751718 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 01:43:46.751718 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 01:43:46.751718 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application with Lock( Aug 30 01:43:46.751718 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 01:43:46.751718 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._enter() Aug 30 01:43:46.751718 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 01:43:46.751718 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 01:43:46.751718 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 01:43:46.751718 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.creator() Aug 30 01:43:46.752693 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 01:43:46.752693 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application created_value = creator( Aug 30 01:43:46.752693 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 30 01:43:46.752693 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 30 01:43:46.752693 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 30 01:43:46.752693 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 30 01:43:46.752693 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 30 01:43:46.752693 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 30 01:43:46.752693 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Aug 30 01:43:46.752693 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Aug 30 01:43:46.753797 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 402/793] 10.222.0.27 () {56 vars in 1117 bytes} [Tue Aug 30 01:43:46 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 248 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 01:43:46.760507 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8366a98c-f98f-4a4b-b916-ac3275d9318e None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:46.827207 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c083e5b3-0117-4f2c-b4a5-332e1ddb7145 None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:46.894310 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4d779ed-995d-43ce-840c-793620bba247 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:46.894950 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4d779ed-995d-43ce-840c-793620bba247 None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:46.895224 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4d779ed-995d-43ce-840c-793620bba247 None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:46.895490 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4d779ed-995d-43ce-840c-793620bba247 None admin] PATH_INFO: `/v3/users/swift` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:46.906596 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4d779ed-995d-43ce-840c-793620bba247 None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:46.906948 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:43:46.908792 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4d779ed-995d-43ce-840c-793620bba247 None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:46.937589 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application [None req-d4d779ed-995d-43ce-840c-793620bba247 None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Aug 30 01:43:46.937589 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 01:43:46.937589 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 01:43:46.937589 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 01:43:46.937589 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 01:43:46.937589 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 01:43:46.937589 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 01:43:46.937589 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 01:43:46.937589 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 01:43:46.937589 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 01:43:46.937589 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 01:43:46.937589 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 01:43:46.937589 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 30 01:43:46.937589 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 30 01:43:46.937589 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 30 01:43:46.937589 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 30 01:43:46.938595 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 01:43:46.938595 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 01:43:46.938595 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 30 01:43:46.938595 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 30 01:43:46.938595 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 30 01:43:46.938595 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 30 01:43:46.938595 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 01:43:46.938595 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 01:43:46.938595 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 01:43:46.938595 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 01:43:46.938595 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 01:43:46.938595 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application with Lock( Aug 30 01:43:46.938595 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 01:43:46.938595 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._enter() Aug 30 01:43:46.938595 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 01:43:46.938595 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 01:43:46.938595 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 01:43:46.938595 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.creator() Aug 30 01:43:46.939524 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 01:43:46.939524 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application created_value = creator( Aug 30 01:43:46.939524 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 30 01:43:46.939524 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 30 01:43:46.939524 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 30 01:43:46.939524 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 30 01:43:46.939524 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 30 01:43:46.939524 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 30 01:43:46.939524 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Aug 30 01:43:46.939524 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application Aug 30 01:43:46.939524 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 392/794] 10.222.0.27 () {56 vars in 1096 bytes} [Tue Aug 30 01:43:46 2022] GET /identity/v3/users/swift => generated 83 bytes in 328 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 30 01:43:46.946900 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4d779ed-995d-43ce-840c-793620bba247 None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:46.960289 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8366a98c-f98f-4a4b-b916-ac3275d9318e None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:46.982934 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:43:47.024729 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9507295a-c7b9-4e87-adc4-14a21e5474db None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:47.025770 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9507295a-c7b9-4e87-adc4-14a21e5474db None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:47.026271 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9507295a-c7b9-4e87-adc4-14a21e5474db None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:47.026745 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9507295a-c7b9-4e87-adc4-14a21e5474db None admin] PATH_INFO: `/v3/users` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:47.027693 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9507295a-c7b9-4e87-adc4-14a21e5474db None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:43:47.028314 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9507295a-c7b9-4e87-adc4-14a21e5474db None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:47.030969 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9507295a-c7b9-4e87-adc4-14a21e5474db None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:47.070745 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 403/795] 10.222.0.27 () {56 vars in 1149 bytes} [Tue Aug 30 01:43:46 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 311 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:43:47.077602 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9507295a-c7b9-4e87-adc4-14a21e5474db None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:47.196849 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9507295a-c7b9-4e87-adc4-14a21e5474db None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:47.224997 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4d779ed-995d-43ce-840c-793620bba247 None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:47.229914 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:43:47.238162 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10488cff-b849-4222-9804-b22b01f86c29 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:47.239003 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10488cff-b849-4222-9804-b22b01f86c29 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:47.239428 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10488cff-b849-4222-9804-b22b01f86c29 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:47.240035 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10488cff-b849-4222-9804-b22b01f86c29 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:47.260270 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10488cff-b849-4222-9804-b22b01f86c29 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:47.262017 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10488cff-b849-4222-9804-b22b01f86c29 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:47.278501 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application [None req-10488cff-b849-4222-9804-b22b01f86c29 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 01:43:47.278501 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 01:43:47.278501 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 01:43:47.278501 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 01:43:47.278501 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 01:43:47.278501 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 01:43:47.278501 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 01:43:47.278501 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 01:43:47.278501 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 01:43:47.278501 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 01:43:47.278501 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 01:43:47.278501 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 01:43:47.278501 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 30 01:43:47.278501 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 30 01:43:47.278501 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 30 01:43:47.278501 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 30 01:43:47.279676 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 01:43:47.279676 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 01:43:47.279676 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 01:43:47.279676 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 01:43:47.279676 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 01:43:47.279676 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 01:43:47.279676 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 01:43:47.279676 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application with Lock( Aug 30 01:43:47.279676 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 01:43:47.279676 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._enter() Aug 30 01:43:47.279676 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 01:43:47.279676 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 01:43:47.279676 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 01:43:47.279676 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.creator() Aug 30 01:43:47.279676 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 01:43:47.279676 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application created_value = creator( Aug 30 01:43:47.279676 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 30 01:43:47.280571 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 30 01:43:47.280571 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 01:43:47.280571 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Aug 30 01:43:47.281712 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-894e5773-868c-412d-b4b3-2904971d3cc9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:47.282410 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-894e5773-868c-412d-b4b3-2904971d3cc9 None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:47.282828 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-894e5773-868c-412d-b4b3-2904971d3cc9 None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:47.283231 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-894e5773-868c-412d-b4b3-2904971d3cc9 None admin] PATH_INFO: `/v3/users` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:47.283396 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 404/796] 10.222.0.27 () {56 vars in 1108 bytes} [Tue Aug 30 01:43:47 2022] GET /identity/v3/domains/Default => generated 87 bytes in 207 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 30 01:43:47.288448 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-894e5773-868c-412d-b4b3-2904971d3cc9 None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=81578) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:43:47.288448 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-894e5773-868c-412d-b4b3-2904971d3cc9 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:47.288448 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-894e5773-868c-412d-b4b3-2904971d3cc9 None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:47.289918 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a996fce-d396-4127-8d2a-01f179a74ffd None None] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:47.290367 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a996fce-d396-4127-8d2a-01f179a74ffd None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:47.290769 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a996fce-d396-4127-8d2a-01f179a74ffd None None] PATH_INFO: `/` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:47.293252 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 405/797] 10.222.0.27 () {54 vars in 927 bytes} [Tue Aug 30 01:43:47 2022] GET /identity => generated 270 bytes in 9 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 01:43:47.294207 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a996fce-d396-4127-8d2a-01f179a74ffd None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:47.305528 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 393/798] 10.222.0.27 () {56 vars in 1135 bytes} [Tue Aug 30 01:43:46 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 359 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 30 01:43:47.306865 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a12e4f3-27f7-4f83-8c80-a695e7abb608 None None] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:47.307153 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a12e4f3-27f7-4f83-8c80-a695e7abb608 None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:47.307428 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a12e4f3-27f7-4f83-8c80-a695e7abb608 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:47.330770 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.auth.core [None req-9a12e4f3-27f7-4f83-8c80-a695e7abb608 None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81578) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 01:43:47.332687 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-9a12e4f3-27f7-4f83-8c80-a695e7abb608 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:43:47.406979 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 394/799] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Aug 30 01:43:47 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 30 01:43:47.408458 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a12e4f3-27f7-4f83-8c80-a695e7abb608 None None] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:47.412141 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:43:47.413094 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a996fce-d396-4127-8d2a-01f179a74ffd None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:47.450992 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9db65e4f-38fe-41e1-8625-f4fd0587d78f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:47.451969 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9db65e4f-38fe-41e1-8625-f4fd0587d78f None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:47.452470 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9db65e4f-38fe-41e1-8625-f4fd0587d78f None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:47.452938 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9db65e4f-38fe-41e1-8625-f4fd0587d78f None admin] PATH_INFO: `/v3/domains` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:47.453935 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9db65e4f-38fe-41e1-8625-f4fd0587d78f None admin] RBAC: Adding query filter params (name=Default) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:43:47.454438 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9db65e4f-38fe-41e1-8625-f4fd0587d78f None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:47.456199 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9db65e4f-38fe-41e1-8625-f4fd0587d78f None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:47.479762 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 406/800] 10.222.0.27 () {56 vars in 1109 bytes} [Tue Aug 30 01:43:47 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 186 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:43:47.485370 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9db65e4f-38fe-41e1-8625-f4fd0587d78f None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:47.548295 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a12e4f3-27f7-4f83-8c80-a695e7abb608 None None] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:47.588387 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c689c7a1-ca5f-43ca-9cc3-f63607976ee8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:47.589028 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c689c7a1-ca5f-43ca-9cc3-f63607976ee8 None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:47.589310 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c689c7a1-ca5f-43ca-9cc3-f63607976ee8 None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:47.589629 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c689c7a1-ca5f-43ca-9cc3-f63607976ee8 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:47.596178 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:43:47.606330 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c689c7a1-ca5f-43ca-9cc3-f63607976ee8 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:47.608261 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c689c7a1-ca5f-43ca-9cc3-f63607976ee8 None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:47.627932 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application [None req-c689c7a1-ca5f-43ca-9cc3-f63607976ee8 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 01:43:47.627932 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 01:43:47.627932 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 01:43:47.627932 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 01:43:47.627932 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 01:43:47.627932 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 01:43:47.627932 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 01:43:47.627932 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 01:43:47.627932 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 01:43:47.627932 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 01:43:47.627932 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 01:43:47.627932 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 01:43:47.627932 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 30 01:43:47.627932 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 30 01:43:47.627932 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 30 01:43:47.627932 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 30 01:43:47.628946 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 01:43:47.628946 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 01:43:47.628946 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 01:43:47.628946 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 01:43:47.628946 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 01:43:47.628946 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 01:43:47.628946 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 01:43:47.628946 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application with Lock( Aug 30 01:43:47.628946 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 01:43:47.628946 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._enter() Aug 30 01:43:47.628946 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 01:43:47.628946 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 01:43:47.628946 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 01:43:47.628946 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.creator() Aug 30 01:43:47.628946 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 01:43:47.628946 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application created_value = creator( Aug 30 01:43:47.628946 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 30 01:43:47.629883 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 30 01:43:47.629883 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 01:43:47.629883 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application Aug 30 01:43:47.632772 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 395/801] 10.222.0.27 () {56 vars in 1108 bytes} [Tue Aug 30 01:43:47 2022] GET /identity/v3/domains/Default => generated 87 bytes in 225 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 01:43:47.638515 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c689c7a1-ca5f-43ca-9cc3-f63607976ee8 None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:47.718157 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9db65e4f-38fe-41e1-8625-f4fd0587d78f None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:47.741425 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:43:47.757404 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a28e8612-e2ba-44cb-bb4b-32e6181d2468 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:47.758472 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a28e8612-e2ba-44cb-bb4b-32e6181d2468 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:47.759010 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a28e8612-e2ba-44cb-bb4b-32e6181d2468 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:47.759328 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a28e8612-e2ba-44cb-bb4b-32e6181d2468 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:47.766553 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a28e8612-e2ba-44cb-bb4b-32e6181d2468 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:47.768102 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a28e8612-e2ba-44cb-bb4b-32e6181d2468 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:47.780857 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application [None req-a28e8612-e2ba-44cb-bb4b-32e6181d2468 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 30 01:43:47.780857 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 01:43:47.780857 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 01:43:47.780857 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 01:43:47.780857 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 01:43:47.780857 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 01:43:47.780857 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 01:43:47.780857 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 01:43:47.780857 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 01:43:47.780857 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 01:43:47.780857 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 01:43:47.780857 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 01:43:47.780857 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 30 01:43:47.780857 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 30 01:43:47.780857 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 30 01:43:47.781843 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 30 01:43:47.781843 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 01:43:47.781843 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 01:43:47.781843 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 01:43:47.781843 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 01:43:47.781843 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 01:43:47.781843 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 01:43:47.781843 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 01:43:47.781843 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application with Lock( Aug 30 01:43:47.781843 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 01:43:47.781843 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._enter() Aug 30 01:43:47.781843 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 01:43:47.781843 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 01:43:47.781843 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 01:43:47.781843 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.creator() Aug 30 01:43:47.781843 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 01:43:47.781843 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application created_value = creator( Aug 30 01:43:47.782762 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 30 01:43:47.782762 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 30 01:43:47.782762 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 30 01:43:47.782762 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 30 01:43:47.782762 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 30 01:43:47.782762 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 30 01:43:47.782762 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 30 01:43:47.782762 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Aug 30 01:43:47.787309 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 407/802] 10.222.0.27 () {56 vars in 1111 bytes} [Tue Aug 30 01:43:47 2022] GET /identity/v3/projects/service => generated 88 bytes in 297 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 01:43:47.788530 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a28e8612-e2ba-44cb-bb4b-32e6181d2468 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:47.795449 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c689c7a1-ca5f-43ca-9cc3-f63607976ee8 None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:47.832226 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d8170e1-74fb-4ee5-af37-9969675b73aa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:47.833159 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d8170e1-74fb-4ee5-af37-9969675b73aa None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:47.834371 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d8170e1-74fb-4ee5-af37-9969675b73aa None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:47.834889 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d8170e1-74fb-4ee5-af37-9969675b73aa None admin] PATH_INFO: `/v3/domains` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:47.835921 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d8170e1-74fb-4ee5-af37-9969675b73aa None admin] RBAC: Adding query filter params (name=Default) {{(pid=81578) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:43:47.836547 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d8170e1-74fb-4ee5-af37-9969675b73aa None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:47.838223 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d8170e1-74fb-4ee5-af37-9969675b73aa None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:47.851980 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 396/803] 10.222.0.27 () {56 vars in 1109 bytes} [Tue Aug 30 01:43:47 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 214 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 30 01:43:47.853180 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d8170e1-74fb-4ee5-af37-9969675b73aa None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:47.893858 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a28e8612-e2ba-44cb-bb4b-32e6181d2468 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:47.897892 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:43:47.933851 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6b5ab02-4135-4951-bc95-7863a2573bc0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:47.934485 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6b5ab02-4135-4951-bc95-7863a2573bc0 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:47.934767 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6b5ab02-4135-4951-bc95-7863a2573bc0 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:47.935042 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6b5ab02-4135-4951-bc95-7863a2573bc0 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:47.949013 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6b5ab02-4135-4951-bc95-7863a2573bc0 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 30 01:43:47.955609 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application [None req-b6b5ab02-4135-4951-bc95-7863a2573bc0 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 30 01:43:47.955609 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 01:43:47.955609 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 01:43:47.955609 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 01:43:47.955609 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 01:43:47.955609 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 01:43:47.955609 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 01:43:47.955609 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 01:43:47.955609 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 01:43:47.955609 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 01:43:47.955609 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 01:43:47.955609 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 01:43:47.955609 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 30 01:43:47.955609 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 30 01:43:47.955609 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 30 01:43:47.955609 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 30 01:43:47.956577 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 30 01:43:47.956577 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 30 01:43:47.956577 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 30 01:43:47.956577 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 30 01:43:47.956577 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 01:43:47.956577 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 01:43:47.956577 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 01:43:47.956577 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 01:43:47.956577 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 01:43:47.956577 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 01:43:47.956577 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 01:43:47.956577 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application with Lock( Aug 30 01:43:47.956577 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 01:43:47.956577 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._enter() Aug 30 01:43:47.956577 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 01:43:47.956577 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 01:43:47.957446 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 01:43:47.957446 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.creator() Aug 30 01:43:47.957446 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 01:43:47.957446 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application created_value = creator( Aug 30 01:43:47.957446 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 30 01:43:47.957446 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 30 01:43:47.957446 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 30 01:43:47.957446 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 30 01:43:47.957446 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 30 01:43:47.957446 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 30 01:43:47.957446 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 30 01:43:47.957446 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Aug 30 01:43:47.957446 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 408/804] 10.222.0.27 () {56 vars in 1096 bytes} [Tue Aug 30 01:43:47 2022] GET /identity/v3/roles/admin => generated 83 bytes in 169 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 01:43:47.958200 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6b5ab02-4135-4951-bc95-7863a2573bc0 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:47.967717 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d8170e1-74fb-4ee5-af37-9969675b73aa None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:47.990721 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:43:48.011321 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d411b8e-8177-4205-bcf8-c70218fd54d6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:48.011921 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d411b8e-8177-4205-bcf8-c70218fd54d6 None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:48.012195 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d411b8e-8177-4205-bcf8-c70218fd54d6 None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:48.012460 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d411b8e-8177-4205-bcf8-c70218fd54d6 None admin] PATH_INFO: `/v3/projects` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:48.013150 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d411b8e-8177-4205-bcf8-c70218fd54d6 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81578) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:43:48.013449 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d411b8e-8177-4205-bcf8-c70218fd54d6 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:48.014878 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d411b8e-8177-4205-bcf8-c70218fd54d6 None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:48.031458 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 397/805] 10.222.0.27 () {56 vars in 1148 bytes} [Tue Aug 30 01:43:47 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:43:48.031458 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d411b8e-8177-4205-bcf8-c70218fd54d6 None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:48.071431 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6b5ab02-4135-4951-bc95-7863a2573bc0 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:48.109363 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4da64d9f-dd1a-4779-b4e8-7b2f77534fb9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:48.111339 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4da64d9f-dd1a-4779-b4e8-7b2f77534fb9 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:48.111339 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4da64d9f-dd1a-4779-b4e8-7b2f77534fb9 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:48.111339 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4da64d9f-dd1a-4779-b4e8-7b2f77534fb9 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:48.120344 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4da64d9f-dd1a-4779-b4e8-7b2f77534fb9 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:48.122260 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4da64d9f-dd1a-4779-b4e8-7b2f77534fb9 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:48.133771 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application [None req-4da64d9f-dd1a-4779-b4e8-7b2f77534fb9 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 30 01:43:48.133771 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 01:43:48.133771 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 01:43:48.133771 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 01:43:48.133771 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 01:43:48.133771 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 01:43:48.133771 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 01:43:48.133771 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 01:43:48.133771 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 01:43:48.133771 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 01:43:48.133771 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 01:43:48.133771 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 01:43:48.133771 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 30 01:43:48.133771 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 30 01:43:48.133771 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 30 01:43:48.134865 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 30 01:43:48.134865 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 01:43:48.134865 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 01:43:48.134865 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 01:43:48.134865 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 01:43:48.134865 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 01:43:48.134865 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 01:43:48.134865 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 01:43:48.134865 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application with Lock( Aug 30 01:43:48.134865 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 01:43:48.134865 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._enter() Aug 30 01:43:48.134865 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 01:43:48.134865 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 01:43:48.134865 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 01:43:48.134865 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.creator() Aug 30 01:43:48.134865 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 01:43:48.134865 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application created_value = creator( Aug 30 01:43:48.135996 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 30 01:43:48.135996 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 30 01:43:48.135996 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 30 01:43:48.135996 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 30 01:43:48.135996 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 30 01:43:48.135996 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 30 01:43:48.135996 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 30 01:43:48.135996 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Aug 30 01:43:48.141551 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 409/806] 10.222.0.27 () {56 vars in 1111 bytes} [Tue Aug 30 01:43:47 2022] GET /identity/v3/projects/service => generated 88 bytes in 179 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 01:43:48.141551 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20a76718-121e-4b1d-947c-f09e0945d462 None None] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:48.141551 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20a76718-121e-4b1d-947c-f09e0945d462 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:48.141551 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20a76718-121e-4b1d-947c-f09e0945d462 None None] PATH_INFO: `/` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:48.144054 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d411b8e-8177-4205-bcf8-c70218fd54d6 None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:48.144172 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 410/807] 10.222.0.27 () {54 vars in 927 bytes} [Tue Aug 30 01:43:48 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 01:43:48.144873 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20a76718-121e-4b1d-947c-f09e0945d462 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:48.168171 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:43:48.184205 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25abb2d3-a115-4333-b3fd-5c2727d28309 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:48.184813 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25abb2d3-a115-4333-b3fd-5c2727d28309 None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:48.185090 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25abb2d3-a115-4333-b3fd-5c2727d28309 None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:48.185364 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25abb2d3-a115-4333-b3fd-5c2727d28309 None admin] PATH_INFO: `/v3/roles` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:48.186058 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25abb2d3-a115-4333-b3fd-5c2727d28309 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81578) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:43:48.186358 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25abb2d3-a115-4333-b3fd-5c2727d28309 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:48.188345 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25abb2d3-a115-4333-b3fd-5c2727d28309 None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:48.201809 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 398/808] 10.222.0.27 () {56 vars in 1099 bytes} [Tue Aug 30 01:43:48 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:43:48.203282 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25abb2d3-a115-4333-b3fd-5c2727d28309 None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:48.244593 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:43:48.250570 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20a76718-121e-4b1d-947c-f09e0945d462 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:48.284033 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4330e829-33c5-43d4-8174-dddbd9728c1a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:48.284688 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4330e829-33c5-43d4-8174-dddbd9728c1a None admin] REQUEST_METHOD: `PUT` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:48.284965 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4330e829-33c5-43d4-8174-dddbd9728c1a None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:48.285245 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4330e829-33c5-43d4-8174-dddbd9728c1a None admin] PATH_INFO: `/v3/projects/4562f3951fbe465280073e60a11e44e2/users/121c374d49a64335ab74a431cfbc1320/roles/aa59b0c9009a480994d9df81a5f8d404` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:48.300870 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4330e829-33c5-43d4-8174-dddbd9728c1a None admin] RBAC: Authorizing `identity:create_grant(project_id=4562f3951fbe465280073e60a11e44e2, user_id=121c374d49a64335ab74a431cfbc1320, role_id=aa59b0c9009a480994d9df81a5f8d404)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:48.302446 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4330e829-33c5-43d4-8174-dddbd9728c1a None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:48.303080 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25abb2d3-a115-4333-b3fd-5c2727d28309 None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:48.335400 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f71f5807-7fb6-4dc5-872b-1e955c4cb87a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:48.335988 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f71f5807-7fb6-4dc5-872b-1e955c4cb87a None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:48.336262 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f71f5807-7fb6-4dc5-872b-1e955c4cb87a None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:48.336527 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f71f5807-7fb6-4dc5-872b-1e955c4cb87a None admin] PATH_INFO: `/v3/projects` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:48.337206 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f71f5807-7fb6-4dc5-872b-1e955c4cb87a None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81578) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:43:48.337531 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f71f5807-7fb6-4dc5-872b-1e955c4cb87a None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:48.338919 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f71f5807-7fb6-4dc5-872b-1e955c4cb87a None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:48.351086 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 399/809] 10.222.0.27 () {56 vars in 1148 bytes} [Tue Aug 30 01:43:48 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:43:48.352589 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faf0059e-3246-4dc6-b3ad-850ce4e04d0a None None] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:48.352898 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faf0059e-3246-4dc6-b3ad-850ce4e04d0a None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:48.353167 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faf0059e-3246-4dc6-b3ad-850ce4e04d0a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:48.370065 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.auth.core [None req-faf0059e-3246-4dc6-b3ad-850ce4e04d0a None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81578) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 01:43:48.371159 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-faf0059e-3246-4dc6-b3ad-850ce4e04d0a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:43:48.393868 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 411/810] 10.222.0.27 () {58 vars in 1439 bytes} [Tue Aug 30 01:43:48 2022] PUT /identity/v3/projects/4562f3951fbe465280073e60a11e44e2/users/121c374d49a64335ab74a431cfbc1320/roles/aa59b0c9009a480994d9df81a5f8d404 => generated 0 bytes in 249 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 01:43:48.394682 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4330e829-33c5-43d4-8174-dddbd9728c1a None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:48.452396 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 400/811] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Aug 30 01:43:48 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 30 01:43:48.453898 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fb18bed-841a-4cea-965e-743a00f1fd4b None None] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:48.454184 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fb18bed-841a-4cea-965e-743a00f1fd4b None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:48.454451 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fb18bed-841a-4cea-965e-743a00f1fd4b None None] PATH_INFO: `/` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:48.457499 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 401/812] 10.222.0.27 () {54 vars in 927 bytes} [Tue Aug 30 01:43:48 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 01:43:48.457499 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fb18bed-841a-4cea-965e-743a00f1fd4b None None] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:48.494321 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:43:48.497709 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:43:48.539117 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4330e829-33c5-43d4-8174-dddbd9728c1a None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:48.588956 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fb18bed-841a-4cea-965e-743a00f1fd4b None None] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:48.590177 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bff6ae4d-7b8b-4523-af60-8143428fa6c7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:48.590878 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bff6ae4d-7b8b-4523-af60-8143428fa6c7 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:48.591223 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bff6ae4d-7b8b-4523-af60-8143428fa6c7 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:48.591561 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bff6ae4d-7b8b-4523-af60-8143428fa6c7 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:48.610275 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bff6ae4d-7b8b-4523-af60-8143428fa6c7 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:48.612157 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bff6ae4d-7b8b-4523-af60-8143428fa6c7 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:48.631073 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fbafce2-37aa-4c9d-b7a6-365d6c0ac88a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:48.632016 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fbafce2-37aa-4c9d-b7a6-365d6c0ac88a None admin] REQUEST_METHOD: `PUT` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:48.632477 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fbafce2-37aa-4c9d-b7a6-365d6c0ac88a None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:48.632581 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application [None req-bff6ae4d-7b8b-4523-af60-8143428fa6c7 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 01:43:48.632581 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 01:43:48.632581 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 01:43:48.632581 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 01:43:48.632581 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 01:43:48.632581 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 01:43:48.632581 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 01:43:48.632581 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 01:43:48.632581 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 01:43:48.632581 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 01:43:48.632581 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 01:43:48.632581 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 01:43:48.632581 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 30 01:43:48.632581 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 30 01:43:48.632581 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 30 01:43:48.632581 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 30 01:43:48.633444 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 01:43:48.633444 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 01:43:48.633444 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 01:43:48.633444 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 01:43:48.633444 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 01:43:48.633444 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 01:43:48.633444 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 01:43:48.633444 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application with Lock( Aug 30 01:43:48.633444 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 01:43:48.633444 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._enter() Aug 30 01:43:48.633444 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 01:43:48.633444 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 01:43:48.633444 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 01:43:48.633444 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.creator() Aug 30 01:43:48.633444 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 01:43:48.633444 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application created_value = creator( Aug 30 01:43:48.633444 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 30 01:43:48.634907 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 30 01:43:48.634907 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 01:43:48.634907 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Aug 30 01:43:48.634907 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 412/813] 10.222.0.27 () {56 vars in 1108 bytes} [Tue Aug 30 01:43:48 2022] GET /identity/v3/domains/Default => generated 87 bytes in 239 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 01:43:48.635673 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fbafce2-37aa-4c9d-b7a6-365d6c0ac88a None admin] PATH_INFO: `/v3/projects/4562f3951fbe465280073e60a11e44e2/users/b4f0008fde424f22bc1f7307cf422774/roles/ead5fac86a0747ea8823a82efcea8ed7` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:48.635776 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26f8e17b-28a7-4a43-96bc-86db45d3784f None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:48.636067 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26f8e17b-28a7-4a43-96bc-86db45d3784f None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:48.636404 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26f8e17b-28a7-4a43-96bc-86db45d3784f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:48.638757 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fbafce2-37aa-4c9d-b7a6-365d6c0ac88a None admin] RBAC: Authorizing `identity:create_grant(project_id=4562f3951fbe465280073e60a11e44e2, user_id=b4f0008fde424f22bc1f7307cf422774, role_id=ead5fac86a0747ea8823a82efcea8ed7)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:48.640407 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fbafce2-37aa-4c9d-b7a6-365d6c0ac88a None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:48.656149 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-26f8e17b-28a7-4a43-96bc-86db45d3784f None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 01:43:48.657501 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-26f8e17b-28a7-4a43-96bc-86db45d3784f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:43:48.735376 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 413/814] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Aug 30 01:43:48 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 30 01:43:48.736225 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26f8e17b-28a7-4a43-96bc-86db45d3784f None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:48.754373 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 402/815] 10.222.0.27 () {58 vars in 1439 bytes} [Tue Aug 30 01:43:48 2022] PUT /identity/v3/projects/4562f3951fbe465280073e60a11e44e2/users/b4f0008fde424f22bc1f7307cf422774/roles/ead5fac86a0747ea8823a82efcea8ed7 => generated 0 bytes in 290 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 01:43:48.757132 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fbafce2-37aa-4c9d-b7a6-365d6c0ac88a None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:48.845146 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:43:48.900700 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26f8e17b-28a7-4a43-96bc-86db45d3784f None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:48.937472 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6688ac14-90d3-47a8-9b65-eb2b1a18890c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:48.938421 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6688ac14-90d3-47a8-9b65-eb2b1a18890c None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:48.938897 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6688ac14-90d3-47a8-9b65-eb2b1a18890c None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:48.939446 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6688ac14-90d3-47a8-9b65-eb2b1a18890c None admin] PATH_INFO: `/v3/domains` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:48.940368 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6688ac14-90d3-47a8-9b65-eb2b1a18890c None admin] RBAC: Adding query filter params (name=Default) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:43:48.940896 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6688ac14-90d3-47a8-9b65-eb2b1a18890c None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:48.942488 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6688ac14-90d3-47a8-9b65-eb2b1a18890c None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:48.949866 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:43:48.955327 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 414/816] 10.222.0.27 () {56 vars in 1109 bytes} [Tue Aug 30 01:43:48 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 219 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:43:48.961508 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6688ac14-90d3-47a8-9b65-eb2b1a18890c None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:48.967861 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fbafce2-37aa-4c9d-b7a6-365d6c0ac88a None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:49.008591 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a98a255-75c3-438a-995e-c4732c20b4f0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:49.009227 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a98a255-75c3-438a-995e-c4732c20b4f0 None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:49.009529 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a98a255-75c3-438a-995e-c4732c20b4f0 None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:49.009805 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a98a255-75c3-438a-995e-c4732c20b4f0 None admin] PATH_INFO: `/v3/services/block-storage` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:49.010390 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a98a255-75c3-438a-995e-c4732c20b4f0 None admin] RBAC: Authorizing `identity:get_service(service_id=block-storage)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:49.011805 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a98a255-75c3-438a-995e-c4732c20b4f0 None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:49.022937 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application [None req-2a98a255-75c3-438a-995e-c4732c20b4f0 None admin] Could not find service: block-storage.: keystone.exception.ServiceNotFound: Could not find service: block-storage. Aug 30 01:43:49.022937 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 01:43:49.022937 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Aug 30 01:43:49.022937 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Aug 30 01:43:49.022937 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Aug 30 01:43:49.022937 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Aug 30 01:43:49.022937 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Aug 30 01:43:49.022937 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 30 01:43:49.022937 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Aug 30 01:43:49.022937 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application Aug 30 01:43:49.022937 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 30 01:43:49.022937 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application Aug 30 01:43:49.022937 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 01:43:49.022937 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 01:43:49.022937 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 01:43:49.022937 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 01:43:49.022937 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 01:43:49.023934 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 01:43:49.023934 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 01:43:49.023934 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 01:43:49.023934 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 01:43:49.023934 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 01:43:49.023934 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 01:43:49.023934 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Aug 30 01:43:49.023934 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._get_service(service_id) Aug 30 01:43:49.023934 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Aug 30 01:43:49.023934 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Aug 30 01:43:49.023934 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 01:43:49.023934 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 01:43:49.023934 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 01:43:49.023934 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 01:43:49.023934 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 01:43:49.023934 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 01:43:49.025148 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 01:43:49.025148 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application with Lock( Aug 30 01:43:49.025148 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 01:43:49.025148 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._enter() Aug 30 01:43:49.025148 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 01:43:49.025148 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 01:43:49.025148 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 01:43:49.025148 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.creator() Aug 30 01:43:49.025148 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 01:43:49.025148 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application created_value = creator( Aug 30 01:43:49.025148 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Aug 30 01:43:49.025148 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 30 01:43:49.025148 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Aug 30 01:43:49.025148 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application Aug 30 01:43:49.025148 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 403/817] 10.222.0.27 () {56 vars in 1129 bytes} [Tue Aug 30 01:43:48 2022] GET /identity/v3/services/block-storage => generated 94 bytes in 269 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 01:43:49.030153 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a98a255-75c3-438a-995e-c4732c20b4f0 None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:49.093221 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6688ac14-90d3-47a8-9b65-eb2b1a18890c None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:49.108335 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:43:49.135672 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-664e195d-5acb-4607-8c70-57ef9f0b2a22 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:49.136476 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-664e195d-5acb-4607-8c70-57ef9f0b2a22 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:49.137065 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-664e195d-5acb-4607-8c70-57ef9f0b2a22 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:49.137444 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-664e195d-5acb-4607-8c70-57ef9f0b2a22 None admin] PATH_INFO: `/v3/users/nova` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:49.147887 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-664e195d-5acb-4607-8c70-57ef9f0b2a22 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:49.150024 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-664e195d-5acb-4607-8c70-57ef9f0b2a22 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:49.153591 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a98a255-75c3-438a-995e-c4732c20b4f0 None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:49.164123 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application [None req-664e195d-5acb-4607-8c70-57ef9f0b2a22 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Aug 30 01:43:49.164123 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 01:43:49.164123 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 01:43:49.164123 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 01:43:49.164123 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 01:43:49.164123 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 01:43:49.164123 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 01:43:49.164123 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 01:43:49.164123 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 01:43:49.164123 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 01:43:49.164123 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 01:43:49.164123 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 01:43:49.164123 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 30 01:43:49.164123 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 30 01:43:49.164123 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 30 01:43:49.164123 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 30 01:43:49.165105 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 01:43:49.165105 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 01:43:49.165105 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 30 01:43:49.165105 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 30 01:43:49.165105 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 30 01:43:49.165105 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 30 01:43:49.165105 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 01:43:49.165105 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 01:43:49.165105 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 01:43:49.165105 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 01:43:49.165105 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 01:43:49.165105 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application with Lock( Aug 30 01:43:49.165105 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 01:43:49.165105 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._enter() Aug 30 01:43:49.165105 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 01:43:49.165105 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 01:43:49.165105 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 01:43:49.165105 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.creator() Aug 30 01:43:49.166076 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 01:43:49.166076 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application created_value = creator( Aug 30 01:43:49.166076 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 30 01:43:49.166076 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 30 01:43:49.166076 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 30 01:43:49.166076 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 30 01:43:49.166076 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 30 01:43:49.166076 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 30 01:43:49.166076 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Aug 30 01:43:49.166076 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Aug 30 01:43:49.169605 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 415/818] 10.222.0.27 () {56 vars in 1093 bytes} [Tue Aug 30 01:43:48 2022] GET /identity/v3/users/nova => generated 82 bytes in 209 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 01:43:49.171494 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-664e195d-5acb-4607-8c70-57ef9f0b2a22 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:49.200529 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd8d8441-703c-4331-9790-d799d61bf801 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:49.201137 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd8d8441-703c-4331-9790-d799d61bf801 None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:49.201410 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd8d8441-703c-4331-9790-d799d61bf801 None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:49.201735 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd8d8441-703c-4331-9790-d799d61bf801 None admin] PATH_INFO: `/v3/services` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:49.202624 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd8d8441-703c-4331-9790-d799d61bf801 None admin] RBAC: Adding query filter params (name=block-storage) {{(pid=81578) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:43:49.202928 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd8d8441-703c-4331-9790-d799d61bf801 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:49.204921 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd8d8441-703c-4331-9790-d799d61bf801 None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:49.215983 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 404/819] 10.222.0.27 () {56 vars in 1124 bytes} [Tue Aug 30 01:43:49 2022] GET /identity/v3/services?name=block-storage => generated 130 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 30 01:43:49.215983 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd8d8441-703c-4331-9790-d799d61bf801 None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:49.236142 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:43:49.304654 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-664e195d-5acb-4607-8c70-57ef9f0b2a22 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:49.338812 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd8d8441-703c-4331-9790-d799d61bf801 None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:49.344978 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5a8cf0e-838a-48f0-9a31-337deb53bd85 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:49.345716 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5a8cf0e-838a-48f0-9a31-337deb53bd85 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:49.346065 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5a8cf0e-838a-48f0-9a31-337deb53bd85 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:49.346412 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5a8cf0e-838a-48f0-9a31-337deb53bd85 None admin] PATH_INFO: `/v3/services/network` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:49.347172 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5a8cf0e-838a-48f0-9a31-337deb53bd85 None admin] RBAC: Authorizing `identity:get_service(service_id=network)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:49.348601 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5a8cf0e-838a-48f0-9a31-337deb53bd85 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:49.359162 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application [None req-f5a8cf0e-838a-48f0-9a31-337deb53bd85 None admin] Could not find service: network.: keystone.exception.ServiceNotFound: Could not find service: network. Aug 30 01:43:49.359162 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 01:43:49.359162 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Aug 30 01:43:49.359162 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Aug 30 01:43:49.359162 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Aug 30 01:43:49.359162 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Aug 30 01:43:49.359162 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Aug 30 01:43:49.359162 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 30 01:43:49.359162 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Aug 30 01:43:49.359162 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Aug 30 01:43:49.359162 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 30 01:43:49.359162 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Aug 30 01:43:49.359162 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 01:43:49.359162 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 01:43:49.359162 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 01:43:49.359162 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 01:43:49.359162 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 01:43:49.360167 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 01:43:49.360167 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 01:43:49.360167 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 01:43:49.360167 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 01:43:49.360167 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 01:43:49.360167 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 01:43:49.360167 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Aug 30 01:43:49.360167 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_service(service_id) Aug 30 01:43:49.360167 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Aug 30 01:43:49.360167 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Aug 30 01:43:49.360167 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 01:43:49.360167 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 01:43:49.360167 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 01:43:49.360167 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 01:43:49.360167 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 01:43:49.360167 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 01:43:49.361012 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 01:43:49.361012 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application with Lock( Aug 30 01:43:49.361012 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 01:43:49.361012 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._enter() Aug 30 01:43:49.361012 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 01:43:49.361012 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 01:43:49.361012 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 01:43:49.361012 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.creator() Aug 30 01:43:49.361012 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 01:43:49.361012 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application created_value = creator( Aug 30 01:43:49.361012 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Aug 30 01:43:49.361012 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 30 01:43:49.361012 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Aug 30 01:43:49.361012 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Aug 30 01:43:49.365208 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 416/820] 10.222.0.27 () {56 vars in 1111 bytes} [Tue Aug 30 01:43:49 2022] GET /identity/v3/services/network => generated 88 bytes in 195 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 01:43:49.366100 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5a8cf0e-838a-48f0-9a31-337deb53bd85 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:49.383701 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32c98682-00fd-47a2-a6c7-a541483c0b50 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:49.385052 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32c98682-00fd-47a2-a6c7-a541483c0b50 None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:49.385586 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32c98682-00fd-47a2-a6c7-a541483c0b50 None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:49.386088 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32c98682-00fd-47a2-a6c7-a541483c0b50 None admin] PATH_INFO: `/v3/users` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:49.387159 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32c98682-00fd-47a2-a6c7-a541483c0b50 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81578) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:43:49.387736 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32c98682-00fd-47a2-a6c7-a541483c0b50 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:49.389467 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32c98682-00fd-47a2-a6c7-a541483c0b50 None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:49.404427 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:43:49.415511 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 405/821] 10.222.0.27 () {56 vars in 1133 bytes} [Tue Aug 30 01:43:49 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 198 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:43:49.416760 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32c98682-00fd-47a2-a6c7-a541483c0b50 None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:49.477136 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5a8cf0e-838a-48f0-9a31-337deb53bd85 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:49.499734 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:43:49.515817 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44bcb2cd-0867-4afe-8843-45ab1c631f2d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:49.517034 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44bcb2cd-0867-4afe-8843-45ab1c631f2d None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:49.517479 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44bcb2cd-0867-4afe-8843-45ab1c631f2d None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:49.517899 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44bcb2cd-0867-4afe-8843-45ab1c631f2d None admin] PATH_INFO: `/v3/services` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:49.518702 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44bcb2cd-0867-4afe-8843-45ab1c631f2d None admin] RBAC: Adding query filter params (type=block-storage) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:43:49.519148 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44bcb2cd-0867-4afe-8843-45ab1c631f2d None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:49.520703 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44bcb2cd-0867-4afe-8843-45ab1c631f2d None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:49.527055 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 417/822] 10.222.0.27 () {56 vars in 1124 bytes} [Tue Aug 30 01:43:49 2022] GET /identity/v3/services?type=block-storage => generated 367 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:43:49.528170 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44bcb2cd-0867-4afe-8843-45ab1c631f2d None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:49.530556 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32c98682-00fd-47a2-a6c7-a541483c0b50 None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:49.576381 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50aeadde-3de2-4cf6-9c8e-4ef8a23e4e54 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:49.577286 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50aeadde-3de2-4cf6-9c8e-4ef8a23e4e54 None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:49.577802 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50aeadde-3de2-4cf6-9c8e-4ef8a23e4e54 None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:49.578380 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50aeadde-3de2-4cf6-9c8e-4ef8a23e4e54 None admin] PATH_INFO: `/v3/services` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:49.580091 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50aeadde-3de2-4cf6-9c8e-4ef8a23e4e54 None admin] RBAC: Adding query filter params (name=network) {{(pid=81578) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:43:49.580609 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50aeadde-3de2-4cf6-9c8e-4ef8a23e4e54 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:49.582331 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50aeadde-3de2-4cf6-9c8e-4ef8a23e4e54 None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:49.593323 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 406/823] 10.222.0.27 () {56 vars in 1112 bytes} [Tue Aug 30 01:43:49 2022] GET /identity/v3/services?name=network => generated 124 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:43:49.593323 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50aeadde-3de2-4cf6-9c8e-4ef8a23e4e54 None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:49.655642 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:43:49.659122 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44bcb2cd-0867-4afe-8843-45ab1c631f2d None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:49.696434 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50aeadde-3de2-4cf6-9c8e-4ef8a23e4e54 None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:49.697351 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc0fff65-3d0c-4b66-91b3-1fe378b2929e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:49.698346 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc0fff65-3d0c-4b66-91b3-1fe378b2929e None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:49.698944 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc0fff65-3d0c-4b66-91b3-1fe378b2929e None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:49.699419 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc0fff65-3d0c-4b66-91b3-1fe378b2929e None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:49.734429 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-122a72b9-4884-4ac0-bf5b-7612555415bb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:49.735043 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-122a72b9-4884-4ac0-bf5b-7612555415bb None admin] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:49.735412 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-122a72b9-4884-4ac0-bf5b-7612555415bb None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:49.735786 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-122a72b9-4884-4ac0-bf5b-7612555415bb None admin] PATH_INFO: `/v3/endpoints` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:49.736442 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-122a72b9-4884-4ac0-bf5b-7612555415bb None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:49.738062 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-122a72b9-4884-4ac0-bf5b-7612555415bb None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:49.747885 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:43:49.782975 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc0fff65-3d0c-4b66-91b3-1fe378b2929e None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:49.784429 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc0fff65-3d0c-4b66-91b3-1fe378b2929e None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:49.844391 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 407/824] 10.222.0.27 () {60 vars in 1144 bytes} [Tue Aug 30 01:43:49 2022] POST /identity/v3/endpoints => generated 346 bytes in 252 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 01:43:49.844391 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-122a72b9-4884-4ac0-bf5b-7612555415bb None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:49.940995 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:43:49.948100 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 418/825] 10.222.0.27 () {58 vars in 1282 bytes} [Tue Aug 30 01:43:49 2022] GET /identity/v3/auth/tokens => generated 1002 bytes in 419 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Aug 30 01:43:49.952180 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc0fff65-3d0c-4b66-91b3-1fe378b2929e None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:50.067132 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-122a72b9-4884-4ac0-bf5b-7612555415bb None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:50.115004 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6df2a6d3-6760-40f2-be60-cdcd4462fdf7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:50.116198 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6df2a6d3-6760-40f2-be60-cdcd4462fdf7 None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:50.116943 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6df2a6d3-6760-40f2-be60-cdcd4462fdf7 None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:50.117871 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6df2a6d3-6760-40f2-be60-cdcd4462fdf7 None admin] PATH_INFO: `/v3/services` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:50.118973 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6df2a6d3-6760-40f2-be60-cdcd4462fdf7 None admin] RBAC: Adding query filter params (type=network) {{(pid=81578) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:43:50.119473 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6df2a6d3-6760-40f2-be60-cdcd4462fdf7 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:50.121355 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc0fff65-3d0c-4b66-91b3-1fe378b2929e None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:50.122520 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6df2a6d3-6760-40f2-be60-cdcd4462fdf7 None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:50.129504 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 408/826] 10.222.0.27 () {56 vars in 1112 bytes} [Tue Aug 30 01:43:49 2022] GET /identity/v3/services?type=network => generated 350 bytes in 286 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:43:50.137023 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6df2a6d3-6760-40f2-be60-cdcd4462fdf7 None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:50.170193 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f1c49d1-1064-4215-b40a-c2bf6ec06520 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:50.170827 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f1c49d1-1064-4215-b40a-c2bf6ec06520 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:50.171106 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f1c49d1-1064-4215-b40a-c2bf6ec06520 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:50.171375 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f1c49d1-1064-4215-b40a-c2bf6ec06520 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:50.187957 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f1c49d1-1064-4215-b40a-c2bf6ec06520 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:50.189417 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f1c49d1-1064-4215-b40a-c2bf6ec06520 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:50.212590 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application [None req-6f1c49d1-1064-4215-b40a-c2bf6ec06520 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 01:43:50.212590 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 01:43:50.212590 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 01:43:50.212590 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 01:43:50.212590 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 01:43:50.212590 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 01:43:50.212590 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 01:43:50.212590 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 01:43:50.212590 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 01:43:50.212590 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 01:43:50.212590 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 01:43:50.212590 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 01:43:50.212590 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 30 01:43:50.212590 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 30 01:43:50.212590 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 30 01:43:50.212590 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 30 01:43:50.213634 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 01:43:50.213634 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 01:43:50.213634 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 01:43:50.213634 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 01:43:50.213634 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 01:43:50.213634 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 01:43:50.213634 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 01:43:50.213634 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application with Lock( Aug 30 01:43:50.213634 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 01:43:50.213634 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._enter() Aug 30 01:43:50.213634 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 01:43:50.213634 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 01:43:50.213634 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 01:43:50.213634 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.creator() Aug 30 01:43:50.213634 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 01:43:50.213634 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application created_value = creator( Aug 30 01:43:50.213634 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 30 01:43:50.214523 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 30 01:43:50.214523 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 01:43:50.214523 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Aug 30 01:43:50.214523 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 419/827] 10.222.0.27 () {56 vars in 1108 bytes} [Tue Aug 30 01:43:49 2022] GET /identity/v3/domains/Default => generated 87 bytes in 262 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 30 01:43:50.218121 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:INSERT=1,keystone:SELECT=99 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:43:50.219421 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f1c49d1-1064-4215-b40a-c2bf6ec06520 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:50.285142 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6df2a6d3-6760-40f2-be60-cdcd4462fdf7 None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:50.292816 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:43:50.331172 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abe8461d-f3fc-415f-8b95-279290594b7a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:50.332119 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abe8461d-f3fc-415f-8b95-279290594b7a None admin] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:50.332581 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abe8461d-f3fc-415f-8b95-279290594b7a None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:50.333041 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abe8461d-f3fc-415f-8b95-279290594b7a None admin] PATH_INFO: `/v3/endpoints` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:50.334469 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abe8461d-f3fc-415f-8b95-279290594b7a None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:50.336365 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abe8461d-f3fc-415f-8b95-279290594b7a None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:50.367118 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f1c49d1-1064-4215-b40a-c2bf6ec06520 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:50.405020 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80a34105-8fdf-4cea-b74e-55690f0fee6d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:50.405737 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80a34105-8fdf-4cea-b74e-55690f0fee6d None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:50.406033 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80a34105-8fdf-4cea-b74e-55690f0fee6d None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:50.406624 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80a34105-8fdf-4cea-b74e-55690f0fee6d None admin] PATH_INFO: `/v3/domains` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:50.407383 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80a34105-8fdf-4cea-b74e-55690f0fee6d None admin] RBAC: Adding query filter params (name=Default) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:43:50.407713 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80a34105-8fdf-4cea-b74e-55690f0fee6d None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:50.409081 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80a34105-8fdf-4cea-b74e-55690f0fee6d None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:50.422492 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 420/828] 10.222.0.27 () {56 vars in 1109 bytes} [Tue Aug 30 01:43:50 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 203 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:43:50.423388 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e790fd85-a40d-46a6-a093-55debd33009c None None] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:50.423618 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e790fd85-a40d-46a6-a093-55debd33009c None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:50.424020 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e790fd85-a40d-46a6-a093-55debd33009c None None] PATH_INFO: `/` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:50.425502 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 421/829] 10.222.0.27 () {54 vars in 927 bytes} [Tue Aug 30 01:43:50 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 01:43:50.428431 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e790fd85-a40d-46a6-a093-55debd33009c None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:50.444478 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 409/830] 10.222.0.27 () {60 vars in 1144 bytes} [Tue Aug 30 01:43:50 2022] POST /identity/v3/endpoints => generated 337 bytes in 305 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 30 01:43:50.446123 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68956523-1937-4fc1-a670-3d855d50b80c None None] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:50.447430 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68956523-1937-4fc1-a670-3d855d50b80c None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:50.449522 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68956523-1937-4fc1-a670-3d855d50b80c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:50.486623 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.auth.core [None req-68956523-1937-4fc1-a670-3d855d50b80c None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81578) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 01:43:50.488722 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-68956523-1937-4fc1-a670-3d855d50b80c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:43:50.608736 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e790fd85-a40d-46a6-a093-55debd33009c None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:50.620495 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:43:50.626303 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 410/831] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Aug 30 01:43:50 2022] POST /identity/v3/auth/tokens => generated 1161 bytes in 180 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 30 01:43:50.648170 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcd4e411-b9af-48b1-baed-c587b95cd1bc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:50.649257 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcd4e411-b9af-48b1-baed-c587b95cd1bc None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:50.649765 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcd4e411-b9af-48b1-baed-c587b95cd1bc None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:50.650238 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcd4e411-b9af-48b1-baed-c587b95cd1bc None admin] PATH_INFO: `/v3/projects/service` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:50.658000 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dcd4e411-b9af-48b1-baed-c587b95cd1bc None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:50.659725 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dcd4e411-b9af-48b1-baed-c587b95cd1bc None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:50.672784 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application [None req-dcd4e411-b9af-48b1-baed-c587b95cd1bc None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 30 01:43:50.672784 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 01:43:50.672784 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 01:43:50.672784 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 01:43:50.672784 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 01:43:50.672784 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 01:43:50.672784 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 01:43:50.672784 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 01:43:50.672784 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 01:43:50.672784 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 01:43:50.672784 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 01:43:50.672784 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 01:43:50.672784 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 30 01:43:50.672784 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 30 01:43:50.672784 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 30 01:43:50.673803 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 30 01:43:50.673803 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 01:43:50.673803 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 01:43:50.673803 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 01:43:50.673803 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 01:43:50.673803 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 01:43:50.673803 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 01:43:50.673803 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 01:43:50.673803 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application with Lock( Aug 30 01:43:50.673803 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 01:43:50.673803 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._enter() Aug 30 01:43:50.673803 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 01:43:50.673803 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 01:43:50.673803 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 01:43:50.673803 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.creator() Aug 30 01:43:50.673803 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 01:43:50.673803 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application created_value = creator( Aug 30 01:43:50.674727 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 30 01:43:50.674727 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 30 01:43:50.674727 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 30 01:43:50.674727 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 30 01:43:50.674727 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 30 01:43:50.674727 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 30 01:43:50.674727 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 30 01:43:50.674727 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Aug 30 01:43:50.681504 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 422/832] 10.222.0.27 () {56 vars in 1111 bytes} [Tue Aug 30 01:43:50 2022] GET /identity/v3/projects/service => generated 88 bytes in 249 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 30 01:43:50.686596 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68956523-1937-4fc1-a670-3d855d50b80c None None] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:50.735978 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:43:50.805867 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68956523-1937-4fc1-a670-3d855d50b80c None None] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:50.846204 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-475c6564-9a20-4ef4-b74f-5065014d3d76 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:50.847085 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-475c6564-9a20-4ef4-b74f-5065014d3d76 None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:50.847517 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-475c6564-9a20-4ef4-b74f-5065014d3d76 None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:50.847990 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-475c6564-9a20-4ef4-b74f-5065014d3d76 None admin] PATH_INFO: `/v3/projects` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:50.849370 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-475c6564-9a20-4ef4-b74f-5065014d3d76 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81578) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:43:50.849874 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-475c6564-9a20-4ef4-b74f-5065014d3d76 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:50.851905 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-475c6564-9a20-4ef4-b74f-5065014d3d76 None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:50.867178 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 411/833] 10.222.0.27 () {56 vars in 1148 bytes} [Tue Aug 30 01:43:50 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:43:50.872895 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcd4e411-b9af-48b1-baed-c587b95cd1bc None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:50.873493 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1780889b-5068-4e76-a54c-9d05c71793df None None] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:50.873940 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1780889b-5068-4e76-a54c-9d05c71793df None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:50.874216 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1780889b-5068-4e76-a54c-9d05c71793df None None] PATH_INFO: `/` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:50.875645 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 412/834] 10.222.0.27 () {54 vars in 927 bytes} [Tue Aug 30 01:43:50 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 01:43:50.881422 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b01739c3-8bee-484d-a34c-f5662d1931a8 None None] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:50.881863 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b01739c3-8bee-484d-a34c-f5662d1931a8 None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:50.882133 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b01739c3-8bee-484d-a34c-f5662d1931a8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:50.905691 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.auth.core [None req-b01739c3-8bee-484d-a34c-f5662d1931a8 None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81578) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 01:43:50.906887 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-b01739c3-8bee-484d-a34c-f5662d1931a8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:43:50.983768 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 413/835] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Aug 30 01:43:50 2022] POST /identity/v3/auth/tokens => generated 1161 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 30 01:43:50.984629 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b01739c3-8bee-484d-a34c-f5662d1931a8 None None] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:51.016385 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:43:51.021544 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcd4e411-b9af-48b1-baed-c587b95cd1bc None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:51.063138 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f0e8cb4-bf4f-400c-b984-1db037f1479b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:51.063809 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f0e8cb4-bf4f-400c-b984-1db037f1479b None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:51.064093 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f0e8cb4-bf4f-400c-b984-1db037f1479b None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:51.064364 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f0e8cb4-bf4f-400c-b984-1db037f1479b None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:51.065099 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f0e8cb4-bf4f-400c-b984-1db037f1479b None admin] RBAC: Adding query filter params (role.id=ead5fac86a0747ea8823a82efcea8ed7, scope.project.id=4562f3951fbe465280073e60a11e44e2, user.id=d64f2e71361644589bec8a06e6eceed1) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:43:51.065592 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f0e8cb4-bf4f-400c-b984-1db037f1479b None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:51.068542 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f0e8cb4-bf4f-400c-b984-1db037f1479b None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:51.077852 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 423/836] 10.222.0.27 () {56 vars in 1374 bytes} [Tue Aug 30 01:43:50 2022] GET /identity/v3/role_assignments?user.id=d64f2e71361644589bec8a06e6eceed1&scope.project.id=4562f3951fbe465280073e60a11e44e2&role.id=ead5fac86a0747ea8823a82efcea8ed7 => generated 608 bytes in 205 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:43:51.108398 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b01739c3-8bee-484d-a34c-f5662d1931a8 None None] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:51.149157 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85ad9a7f-fabd-4a7d-97fd-85aed369fbd3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:51.150175 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85ad9a7f-fabd-4a7d-97fd-85aed369fbd3 None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:51.150661 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85ad9a7f-fabd-4a7d-97fd-85aed369fbd3 None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:51.151079 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85ad9a7f-fabd-4a7d-97fd-85aed369fbd3 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:51.164148 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85ad9a7f-fabd-4a7d-97fd-85aed369fbd3 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 30 01:43:51.169565 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application [None req-85ad9a7f-fabd-4a7d-97fd-85aed369fbd3 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Aug 30 01:43:51.169565 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 01:43:51.169565 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 01:43:51.169565 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 01:43:51.169565 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 01:43:51.169565 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 01:43:51.169565 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 01:43:51.169565 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 01:43:51.169565 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 01:43:51.169565 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 01:43:51.169565 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 01:43:51.169565 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 01:43:51.169565 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 30 01:43:51.169565 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 30 01:43:51.169565 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 30 01:43:51.169565 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 30 01:43:51.170722 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 30 01:43:51.170722 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 30 01:43:51.170722 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 30 01:43:51.170722 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 30 01:43:51.170722 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 01:43:51.170722 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 01:43:51.170722 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 01:43:51.170722 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 01:43:51.170722 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 01:43:51.170722 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 01:43:51.170722 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 01:43:51.170722 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application with Lock( Aug 30 01:43:51.170722 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 01:43:51.170722 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._enter() Aug 30 01:43:51.170722 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 01:43:51.170722 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 01:43:51.171764 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 01:43:51.171764 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.creator() Aug 30 01:43:51.171764 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 01:43:51.171764 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application created_value = creator( Aug 30 01:43:51.171764 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 30 01:43:51.171764 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 30 01:43:51.171764 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 30 01:43:51.171764 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 30 01:43:51.171764 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 30 01:43:51.171764 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 30 01:43:51.171764 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Aug 30 01:43:51.171764 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application Aug 30 01:43:51.174165 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 414/837] 10.222.0.27 () {56 vars in 1102 bytes} [Tue Aug 30 01:43:50 2022] GET /identity/v3/roles/service => generated 85 bytes in 190 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 01:43:51.182156 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f0e8cb4-bf4f-400c-b984-1db037f1479b None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:51.217742 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:43:51.291700 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85ad9a7f-fabd-4a7d-97fd-85aed369fbd3 None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:51.293304 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f0e8cb4-bf4f-400c-b984-1db037f1479b None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:51.331436 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb51f6ca-ae02-462f-9929-5c94b38357d3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:51.332131 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb51f6ca-ae02-462f-9929-5c94b38357d3 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:51.332511 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb51f6ca-ae02-462f-9929-5c94b38357d3 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:51.332939 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb51f6ca-ae02-462f-9929-5c94b38357d3 None admin] PATH_INFO: `/v3/roles` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:51.334304 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb51f6ca-ae02-462f-9929-5c94b38357d3 None admin] RBAC: Adding query filter params (name=service) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:43:51.334705 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb51f6ca-ae02-462f-9929-5c94b38357d3 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:51.336278 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb51f6ca-ae02-462f-9929-5c94b38357d3 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:51.345902 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 424/838] 10.222.0.27 () {56 vars in 1103 bytes} [Tue Aug 30 01:43:51 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:43:51.352473 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb51f6ca-ae02-462f-9929-5c94b38357d3 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:51.434061 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:43:51.447618 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85ad9a7f-fabd-4a7d-97fd-85aed369fbd3 None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:51.461815 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:43:51.469450 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb51f6ca-ae02-462f-9929-5c94b38357d3 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:51.497191 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74d08475-4311-4302-a2c2-dc391466a4ab None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:51.499347 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74d08475-4311-4302-a2c2-dc391466a4ab None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:51.499801 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74d08475-4311-4302-a2c2-dc391466a4ab None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:51.500216 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74d08475-4311-4302-a2c2-dc391466a4ab None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:51.507378 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be759e51-9c95-4566-b60f-2911d57c8df0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:51.508152 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be759e51-9c95-4566-b60f-2911d57c8df0 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:51.508625 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be759e51-9c95-4566-b60f-2911d57c8df0 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:51.509089 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be759e51-9c95-4566-b60f-2911d57c8df0 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:51.514610 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74d08475-4311-4302-a2c2-dc391466a4ab None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 30 01:43:51.520648 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be759e51-9c95-4566-b60f-2911d57c8df0 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:51.521143 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application [None req-74d08475-4311-4302-a2c2-dc391466a4ab None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 30 01:43:51.521143 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 01:43:51.521143 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 01:43:51.521143 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 01:43:51.521143 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 01:43:51.521143 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 01:43:51.521143 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 01:43:51.521143 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 01:43:51.521143 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 01:43:51.521143 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 01:43:51.521143 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 01:43:51.521143 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 01:43:51.521143 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 30 01:43:51.521143 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 30 01:43:51.521143 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 30 01:43:51.521143 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 30 01:43:51.522231 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 30 01:43:51.522231 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 30 01:43:51.522231 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 30 01:43:51.522231 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 30 01:43:51.522231 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 01:43:51.522231 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 01:43:51.522231 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 01:43:51.522231 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 01:43:51.522231 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 01:43:51.522231 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 01:43:51.522231 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 01:43:51.522231 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application with Lock( Aug 30 01:43:51.522231 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 01:43:51.522231 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._enter() Aug 30 01:43:51.522231 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 01:43:51.522231 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 01:43:51.523231 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 01:43:51.523231 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.creator() Aug 30 01:43:51.523231 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 01:43:51.523231 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application created_value = creator( Aug 30 01:43:51.523231 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 30 01:43:51.523231 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 30 01:43:51.523231 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 30 01:43:51.523231 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 30 01:43:51.523231 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 30 01:43:51.523231 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 30 01:43:51.523231 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 30 01:43:51.523231 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application Aug 30 01:43:51.525054 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be759e51-9c95-4566-b60f-2911d57c8df0 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:51.530243 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 415/839] 10.222.0.27 () {56 vars in 1096 bytes} [Tue Aug 30 01:43:51 2022] GET /identity/v3/roles/admin => generated 83 bytes in 234 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 01:43:51.531223 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74d08475-4311-4302-a2c2-dc391466a4ab None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:51.553415 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application [None req-be759e51-9c95-4566-b60f-2911d57c8df0 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 01:43:51.553415 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 01:43:51.553415 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 01:43:51.553415 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 01:43:51.553415 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 01:43:51.553415 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 01:43:51.553415 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 01:43:51.553415 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 01:43:51.553415 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 01:43:51.553415 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 01:43:51.553415 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 01:43:51.553415 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 01:43:51.553415 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 30 01:43:51.553415 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 30 01:43:51.553415 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 30 01:43:51.553415 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 30 01:43:51.554627 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 01:43:51.554627 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 01:43:51.554627 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 01:43:51.554627 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 01:43:51.554627 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 01:43:51.554627 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 01:43:51.554627 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 01:43:51.554627 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application with Lock( Aug 30 01:43:51.554627 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 01:43:51.554627 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._enter() Aug 30 01:43:51.554627 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 01:43:51.554627 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 01:43:51.554627 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 01:43:51.554627 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.creator() Aug 30 01:43:51.554627 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 01:43:51.554627 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application created_value = creator( Aug 30 01:43:51.554627 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 30 01:43:51.555711 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 30 01:43:51.555711 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 01:43:51.555711 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Aug 30 01:43:51.557811 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 425/840] 10.222.0.27 () {56 vars in 1108 bytes} [Tue Aug 30 01:43:51 2022] GET /identity/v3/domains/Default => generated 87 bytes in 206 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 30 01:43:51.563515 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be759e51-9c95-4566-b60f-2911d57c8df0 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:51.671161 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74d08475-4311-4302-a2c2-dc391466a4ab None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:51.698766 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be759e51-9c95-4566-b60f-2911d57c8df0 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:51.719222 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c417cdcd-54af-4f76-be73-d94de8ca7e1a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:51.719222 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c417cdcd-54af-4f76-be73-d94de8ca7e1a None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:51.719222 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c417cdcd-54af-4f76-be73-d94de8ca7e1a None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:51.719222 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c417cdcd-54af-4f76-be73-d94de8ca7e1a None admin] PATH_INFO: `/v3/roles` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:51.719664 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c417cdcd-54af-4f76-be73-d94de8ca7e1a None admin] RBAC: Adding query filter params (name=admin) {{(pid=81578) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:43:51.719664 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c417cdcd-54af-4f76-be73-d94de8ca7e1a None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:51.723075 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c417cdcd-54af-4f76-be73-d94de8ca7e1a None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:51.736662 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 416/841] 10.222.0.27 () {56 vars in 1099 bytes} [Tue Aug 30 01:43:51 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 206 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:43:51.742847 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c417cdcd-54af-4f76-be73-d94de8ca7e1a None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:51.759045 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:43:51.761481 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1021b745-82dc-4f23-b6e5-f2753643ba4e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:51.762258 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1021b745-82dc-4f23-b6e5-f2753643ba4e None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:51.762683 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1021b745-82dc-4f23-b6e5-f2753643ba4e None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:51.763154 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1021b745-82dc-4f23-b6e5-f2753643ba4e None admin] PATH_INFO: `/v3/domains` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:51.764026 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1021b745-82dc-4f23-b6e5-f2753643ba4e None admin] RBAC: Adding query filter params (name=Default) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:43:51.764574 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1021b745-82dc-4f23-b6e5-f2753643ba4e None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:51.766209 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1021b745-82dc-4f23-b6e5-f2753643ba4e None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:51.781704 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 426/842] 10.222.0.27 () {56 vars in 1109 bytes} [Tue Aug 30 01:43:51 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 216 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:43:51.787295 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1021b745-82dc-4f23-b6e5-f2753643ba4e None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:51.796563 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:43:51.852403 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c417cdcd-54af-4f76-be73-d94de8ca7e1a None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:51.888687 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6da74b73-3bd3-4287-b4ba-1938612916c1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:51.889400 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6da74b73-3bd3-4287-b4ba-1938612916c1 None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:51.889712 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6da74b73-3bd3-4287-b4ba-1938612916c1 None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:51.890101 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6da74b73-3bd3-4287-b4ba-1938612916c1 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:51.902084 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6da74b73-3bd3-4287-b4ba-1938612916c1 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:51.902706 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1021b745-82dc-4f23-b6e5-f2753643ba4e None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:51.903529 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6da74b73-3bd3-4287-b4ba-1938612916c1 None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:51.919333 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application [None req-6da74b73-3bd3-4287-b4ba-1938612916c1 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 01:43:51.919333 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 01:43:51.919333 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 01:43:51.919333 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 01:43:51.919333 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 01:43:51.919333 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 01:43:51.919333 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 01:43:51.919333 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 01:43:51.919333 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 01:43:51.919333 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 01:43:51.919333 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 01:43:51.919333 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 01:43:51.919333 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 30 01:43:51.919333 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 30 01:43:51.919333 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 30 01:43:51.919333 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 30 01:43:51.920503 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 01:43:51.920503 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 01:43:51.920503 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 01:43:51.920503 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 01:43:51.920503 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 01:43:51.920503 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 01:43:51.920503 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 01:43:51.920503 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application with Lock( Aug 30 01:43:51.920503 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 01:43:51.920503 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._enter() Aug 30 01:43:51.920503 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 01:43:51.920503 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 01:43:51.920503 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 01:43:51.920503 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.creator() Aug 30 01:43:51.920503 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 01:43:51.920503 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application created_value = creator( Aug 30 01:43:51.920503 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 30 01:43:51.921556 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 30 01:43:51.921556 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 01:43:51.921556 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application Aug 30 01:43:51.925482 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 417/843] 10.222.0.27 () {56 vars in 1108 bytes} [Tue Aug 30 01:43:51 2022] GET /identity/v3/domains/Default => generated 87 bytes in 178 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 30 01:43:51.926647 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6da74b73-3bd3-4287-b4ba-1938612916c1 None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:51.945695 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4b059dc-4c09-41a5-9f3f-1067c82da5b0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:51.946625 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4b059dc-4c09-41a5-9f3f-1067c82da5b0 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:51.947052 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4b059dc-4c09-41a5-9f3f-1067c82da5b0 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:51.947505 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4b059dc-4c09-41a5-9f3f-1067c82da5b0 None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:51.961577 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4b059dc-4c09-41a5-9f3f-1067c82da5b0 None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:51.963317 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4b059dc-4c09-41a5-9f3f-1067c82da5b0 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:51.982115 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application [None req-d4b059dc-4c09-41a5-9f3f-1067c82da5b0 None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Aug 30 01:43:51.982115 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 01:43:51.982115 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 01:43:51.982115 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 01:43:51.982115 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 01:43:51.982115 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 01:43:51.982115 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 01:43:51.982115 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 01:43:51.982115 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 01:43:51.982115 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 01:43:51.982115 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 01:43:51.982115 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 01:43:51.982115 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 30 01:43:51.982115 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 30 01:43:51.982115 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 30 01:43:51.982115 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 30 01:43:51.983230 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 01:43:51.983230 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 01:43:51.983230 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 30 01:43:51.983230 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 30 01:43:51.983230 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 30 01:43:51.983230 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 30 01:43:51.983230 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 01:43:51.983230 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 01:43:51.983230 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 01:43:51.983230 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 01:43:51.983230 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 01:43:51.983230 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application with Lock( Aug 30 01:43:51.983230 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 01:43:51.983230 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._enter() Aug 30 01:43:51.983230 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 01:43:51.983230 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 01:43:51.983230 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 01:43:51.983230 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.creator() Aug 30 01:43:51.984181 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 01:43:51.984181 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application created_value = creator( Aug 30 01:43:51.984181 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 30 01:43:51.984181 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 30 01:43:51.984181 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 30 01:43:51.984181 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 30 01:43:51.984181 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 30 01:43:51.984181 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 30 01:43:51.984181 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Aug 30 01:43:51.984181 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Aug 30 01:43:51.992430 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 427/844] 10.222.0.27 () {56 vars in 1117 bytes} [Tue Aug 30 01:43:51 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 201 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 01:43:51.993443 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4b059dc-4c09-41a5-9f3f-1067c82da5b0 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:51.999592 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:43:52.075188 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6da74b73-3bd3-4287-b4ba-1938612916c1 None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:52.098032 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:43:52.113356 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60dedee5-2a21-4e3f-a6a2-7dcc11b24ff6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:52.114167 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60dedee5-2a21-4e3f-a6a2-7dcc11b24ff6 None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:52.115008 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60dedee5-2a21-4e3f-a6a2-7dcc11b24ff6 None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:52.115399 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60dedee5-2a21-4e3f-a6a2-7dcc11b24ff6 None admin] PATH_INFO: `/v3/domains` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:52.116359 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60dedee5-2a21-4e3f-a6a2-7dcc11b24ff6 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81578) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:43:52.116668 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60dedee5-2a21-4e3f-a6a2-7dcc11b24ff6 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:52.118089 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60dedee5-2a21-4e3f-a6a2-7dcc11b24ff6 None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:52.136283 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 418/845] 10.222.0.27 () {56 vars in 1109 bytes} [Tue Aug 30 01:43:51 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 205 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:43:52.137214 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60dedee5-2a21-4e3f-a6a2-7dcc11b24ff6 None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:52.141307 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4b059dc-4c09-41a5-9f3f-1067c82da5b0 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:52.180780 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e4e76ed-6803-4e2b-b34b-cd489bae41b1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:52.181665 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e4e76ed-6803-4e2b-b34b-cd489bae41b1 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:52.182131 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e4e76ed-6803-4e2b-b34b-cd489bae41b1 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:52.182604 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e4e76ed-6803-4e2b-b34b-cd489bae41b1 None admin] PATH_INFO: `/v3/users` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:52.183641 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e4e76ed-6803-4e2b-b34b-cd489bae41b1 None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:43:52.184114 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e4e76ed-6803-4e2b-b34b-cd489bae41b1 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:52.185870 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e4e76ed-6803-4e2b-b34b-cd489bae41b1 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:52.209855 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 428/846] 10.222.0.27 () {56 vars in 1149 bytes} [Tue Aug 30 01:43:51 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 217 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:43:52.212551 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e4e76ed-6803-4e2b-b34b-cd489bae41b1 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:52.252757 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60dedee5-2a21-4e3f-a6a2-7dcc11b24ff6 None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:52.295342 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:43:52.307264 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c29cc3f1-46f6-449c-b1bf-4bc033e6fa2c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:52.308067 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c29cc3f1-46f6-449c-b1bf-4bc033e6fa2c None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:52.308561 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c29cc3f1-46f6-449c-b1bf-4bc033e6fa2c None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:52.309105 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c29cc3f1-46f6-449c-b1bf-4bc033e6fa2c None admin] PATH_INFO: `/v3/users/swift` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:52.317235 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c29cc3f1-46f6-449c-b1bf-4bc033e6fa2c None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:52.319702 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c29cc3f1-46f6-449c-b1bf-4bc033e6fa2c None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:52.332519 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e4e76ed-6803-4e2b-b34b-cd489bae41b1 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:52.333127 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application [None req-c29cc3f1-46f6-449c-b1bf-4bc033e6fa2c None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Aug 30 01:43:52.333127 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 01:43:52.333127 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 01:43:52.333127 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 01:43:52.333127 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 01:43:52.333127 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 01:43:52.333127 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 01:43:52.333127 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 01:43:52.333127 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 01:43:52.333127 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 01:43:52.333127 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 01:43:52.333127 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 01:43:52.333127 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 30 01:43:52.333127 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 30 01:43:52.333127 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 30 01:43:52.333127 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 30 01:43:52.334606 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 01:43:52.334606 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 01:43:52.334606 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 30 01:43:52.334606 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 30 01:43:52.334606 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 30 01:43:52.334606 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 30 01:43:52.334606 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 01:43:52.334606 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 01:43:52.334606 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 01:43:52.334606 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 01:43:52.334606 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 01:43:52.334606 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application with Lock( Aug 30 01:43:52.334606 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 01:43:52.334606 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._enter() Aug 30 01:43:52.334606 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 01:43:52.334606 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 01:43:52.334606 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 01:43:52.334606 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.creator() Aug 30 01:43:52.335544 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 01:43:52.335544 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application created_value = creator( Aug 30 01:43:52.335544 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 30 01:43:52.335544 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 30 01:43:52.335544 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 30 01:43:52.335544 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 30 01:43:52.335544 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 30 01:43:52.335544 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 30 01:43:52.335544 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Aug 30 01:43:52.335544 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application Aug 30 01:43:52.340339 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 419/847] 10.222.0.27 () {56 vars in 1096 bytes} [Tue Aug 30 01:43:52 2022] GET /identity/v3/users/swift => generated 83 bytes in 203 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 01:43:52.341724 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04fef217-be4e-46d8-9835-b77dca346e8c None None] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:52.342111 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04fef217-be4e-46d8-9835-b77dca346e8c None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:52.342386 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04fef217-be4e-46d8-9835-b77dca346e8c None None] PATH_INFO: `/` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:52.347713 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 420/848] 10.222.0.27 () {54 vars in 927 bytes} [Tue Aug 30 01:43:52 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 01:43:52.347713 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04fef217-be4e-46d8-9835-b77dca346e8c None None] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:52.362027 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:43:52.378194 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-360badd2-3fe1-4d8f-8e1c-652c0d984dbc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:52.378802 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-360badd2-3fe1-4d8f-8e1c-652c0d984dbc None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:52.379075 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-360badd2-3fe1-4d8f-8e1c-652c0d984dbc None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:52.379344 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-360badd2-3fe1-4d8f-8e1c-652c0d984dbc None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:52.471535 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-360badd2-3fe1-4d8f-8e1c-652c0d984dbc None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:52.473080 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-360badd2-3fe1-4d8f-8e1c-652c0d984dbc None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:52.486915 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04fef217-be4e-46d8-9835-b77dca346e8c None None] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:52.531986 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13b0960a-5353-4713-a596-d78575ec630e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:52.533223 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13b0960a-5353-4713-a596-d78575ec630e None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:52.533681 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13b0960a-5353-4713-a596-d78575ec630e None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:52.534095 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13b0960a-5353-4713-a596-d78575ec630e None admin] PATH_INFO: `/v3/users` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:52.535528 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13b0960a-5353-4713-a596-d78575ec630e None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=81578) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:43:52.535984 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13b0960a-5353-4713-a596-d78575ec630e None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:52.538910 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13b0960a-5353-4713-a596-d78575ec630e None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:52.565122 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 421/849] 10.222.0.27 () {56 vars in 1135 bytes} [Tue Aug 30 01:43:52 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 220 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:43:52.566584 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4334d734-e638-42be-9a28-5fb71244e60b None None] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:52.566873 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4334d734-e638-42be-9a28-5fb71244e60b None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:52.567150 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4334d734-e638-42be-9a28-5fb71244e60b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:52.587729 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.auth.core [None req-4334d734-e638-42be-9a28-5fb71244e60b None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81578) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 01:43:52.588877 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-4334d734-e638-42be-9a28-5fb71244e60b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:43:52.602578 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 429/850] 10.222.0.27 () {58 vars in 1282 bytes} [Tue Aug 30 01:43:52 2022] GET /identity/v3/auth/tokens => generated 1161 bytes in 386 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Aug 30 01:43:52.602578 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-360badd2-3fe1-4d8f-8e1c-652c0d984dbc None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:52.627558 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:43:52.628284 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:43:52.662478 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 422/851] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Aug 30 01:43:52 2022] POST /identity/v3/auth/tokens => generated 1161 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 30 01:43:52.663345 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4334d734-e638-42be-9a28-5fb71244e60b None None] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:52.726149 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-360badd2-3fe1-4d8f-8e1c-652c0d984dbc None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:52.764184 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abf41b46-8a6f-48f8-9100-be68078bf51f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:52.765080 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abf41b46-8a6f-48f8-9100-be68078bf51f None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:52.765585 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abf41b46-8a6f-48f8-9100-be68078bf51f None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:52.765999 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abf41b46-8a6f-48f8-9100-be68078bf51f None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:52.797259 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4334d734-e638-42be-9a28-5fb71244e60b None None] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:52.835720 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f46f3d3f-a34e-493c-a180-55b8e3b6518c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:52.836599 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f46f3d3f-a34e-493c-a180-55b8e3b6518c None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:52.836988 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f46f3d3f-a34e-493c-a180-55b8e3b6518c None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:52.837383 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f46f3d3f-a34e-493c-a180-55b8e3b6518c None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:52.844760 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abf41b46-8a6f-48f8-9100-be68078bf51f None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:52.846497 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abf41b46-8a6f-48f8-9100-be68078bf51f None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:52.848471 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f46f3d3f-a34e-493c-a180-55b8e3b6518c None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:52.850329 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f46f3d3f-a34e-493c-a180-55b8e3b6518c None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:52.867876 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application [None req-f46f3d3f-a34e-493c-a180-55b8e3b6518c None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 01:43:52.867876 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 01:43:52.867876 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 01:43:52.867876 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 01:43:52.867876 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 01:43:52.867876 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 01:43:52.867876 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 01:43:52.867876 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 01:43:52.867876 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 01:43:52.867876 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 01:43:52.867876 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 01:43:52.867876 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 01:43:52.867876 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 30 01:43:52.867876 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 30 01:43:52.867876 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 30 01:43:52.867876 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 30 01:43:52.869064 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 01:43:52.869064 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 01:43:52.869064 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 01:43:52.869064 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 01:43:52.869064 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 01:43:52.869064 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 01:43:52.869064 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 01:43:52.869064 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application with Lock( Aug 30 01:43:52.869064 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 01:43:52.869064 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._enter() Aug 30 01:43:52.869064 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 01:43:52.869064 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 01:43:52.869064 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 01:43:52.869064 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.creator() Aug 30 01:43:52.869064 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 01:43:52.869064 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application created_value = creator( Aug 30 01:43:52.869064 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 30 01:43:52.870221 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 30 01:43:52.870221 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 01:43:52.870221 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application Aug 30 01:43:52.876244 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 423/852] 10.222.0.27 () {56 vars in 1108 bytes} [Tue Aug 30 01:43:52 2022] GET /identity/v3/domains/Default => generated 87 bytes in 208 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 01:43:52.878242 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f46f3d3f-a34e-493c-a180-55b8e3b6518c None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:52.882263 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:43:52.910814 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:43:52.974836 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 430/853] 10.222.0.27 () {58 vars in 1282 bytes} [Tue Aug 30 01:43:52 2022] GET /identity/v3/auth/tokens => generated 1161 bytes in 376 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Aug 30 01:43:52.975756 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abf41b46-8a6f-48f8-9100-be68078bf51f None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:53.000823 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f46f3d3f-a34e-493c-a180-55b8e3b6518c None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:53.034627 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-647f7c99-b44e-4050-b79c-d540e48ff791 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:53.035224 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-647f7c99-b44e-4050-b79c-d540e48ff791 None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:53.035499 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-647f7c99-b44e-4050-b79c-d540e48ff791 None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:53.035764 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-647f7c99-b44e-4050-b79c-d540e48ff791 None admin] PATH_INFO: `/v3/domains` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:53.037035 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-647f7c99-b44e-4050-b79c-d540e48ff791 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81578) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:43:53.037335 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-647f7c99-b44e-4050-b79c-d540e48ff791 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:53.038721 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-647f7c99-b44e-4050-b79c-d540e48ff791 None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:53.055319 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 424/854] 10.222.0.27 () {56 vars in 1109 bytes} [Tue Aug 30 01:43:52 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:43:53.055319 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-647f7c99-b44e-4050-b79c-d540e48ff791 None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:53.091537 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abf41b46-8a6f-48f8-9100-be68078bf51f None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:53.121870 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:43:53.133181 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0649987-0506-4bdf-8a97-068fa792338a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:53.133842 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0649987-0506-4bdf-8a97-068fa792338a None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:53.134121 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0649987-0506-4bdf-8a97-068fa792338a None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:53.134391 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0649987-0506-4bdf-8a97-068fa792338a None admin] PATH_INFO: `/v3/services/volumev3` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:53.141400 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0649987-0506-4bdf-8a97-068fa792338a None admin] RBAC: Authorizing `identity:get_service(service_id=volumev3)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:53.141400 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0649987-0506-4bdf-8a97-068fa792338a None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:53.144937 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application [None req-e0649987-0506-4bdf-8a97-068fa792338a None admin] Could not find service: volumev3.: keystone.exception.ServiceNotFound: Could not find service: volumev3. Aug 30 01:43:53.144937 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 01:43:53.144937 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Aug 30 01:43:53.144937 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Aug 30 01:43:53.144937 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Aug 30 01:43:53.144937 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Aug 30 01:43:53.144937 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Aug 30 01:43:53.144937 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 30 01:43:53.144937 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Aug 30 01:43:53.144937 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Aug 30 01:43:53.144937 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 30 01:43:53.144937 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Aug 30 01:43:53.144937 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 01:43:53.144937 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 01:43:53.144937 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 01:43:53.144937 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 01:43:53.144937 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 01:43:53.146706 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 01:43:53.146706 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 01:43:53.146706 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 01:43:53.146706 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 01:43:53.146706 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 01:43:53.146706 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 01:43:53.146706 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Aug 30 01:43:53.146706 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_service(service_id) Aug 30 01:43:53.146706 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Aug 30 01:43:53.146706 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Aug 30 01:43:53.146706 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 01:43:53.146706 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 01:43:53.146706 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 01:43:53.146706 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 01:43:53.146706 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 01:43:53.146706 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 01:43:53.147997 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 01:43:53.147997 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application with Lock( Aug 30 01:43:53.147997 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 01:43:53.147997 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._enter() Aug 30 01:43:53.147997 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 01:43:53.147997 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 01:43:53.147997 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 01:43:53.147997 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.creator() Aug 30 01:43:53.147997 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 01:43:53.147997 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application created_value = creator( Aug 30 01:43:53.147997 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Aug 30 01:43:53.147997 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 30 01:43:53.147997 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Aug 30 01:43:53.147997 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Aug 30 01:43:53.147997 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 431/855] 10.222.0.27 () {56 vars in 1114 bytes} [Tue Aug 30 01:43:52 2022] GET /identity/v3/services/volumev3 => generated 89 bytes in 171 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 01:43:53.148837 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0649987-0506-4bdf-8a97-068fa792338a None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:53.151032 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:43:53.167066 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-647f7c99-b44e-4050-b79c-d540e48ff791 None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:53.204421 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf78f961-053c-40a5-86a8-6e5d1859d219 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:53.205047 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf78f961-053c-40a5-86a8-6e5d1859d219 None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:53.205326 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf78f961-053c-40a5-86a8-6e5d1859d219 None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:53.205624 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf78f961-053c-40a5-86a8-6e5d1859d219 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:53.216767 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf78f961-053c-40a5-86a8-6e5d1859d219 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:53.218354 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf78f961-053c-40a5-86a8-6e5d1859d219 None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:53.232736 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application [None req-cf78f961-053c-40a5-86a8-6e5d1859d219 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 01:43:53.232736 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 01:43:53.232736 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 01:43:53.232736 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 01:43:53.232736 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 01:43:53.232736 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 01:43:53.232736 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 01:43:53.232736 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 01:43:53.232736 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 01:43:53.232736 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 01:43:53.232736 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 01:43:53.232736 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 01:43:53.232736 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 30 01:43:53.232736 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 30 01:43:53.232736 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 30 01:43:53.232736 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 30 01:43:53.233717 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 01:43:53.233717 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 01:43:53.233717 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 01:43:53.233717 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 01:43:53.233717 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 01:43:53.233717 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 01:43:53.233717 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 01:43:53.233717 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application with Lock( Aug 30 01:43:53.233717 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 01:43:53.233717 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._enter() Aug 30 01:43:53.233717 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 01:43:53.233717 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 01:43:53.233717 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 01:43:53.233717 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.creator() Aug 30 01:43:53.233717 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 01:43:53.233717 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application created_value = creator( Aug 30 01:43:53.233717 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 30 01:43:53.234642 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 30 01:43:53.234642 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 01:43:53.234642 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application Aug 30 01:43:53.234642 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 425/856] 10.222.0.27 () {56 vars in 1108 bytes} [Tue Aug 30 01:43:53 2022] GET /identity/v3/domains/Default => generated 87 bytes in 183 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 01:43:53.234936 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74490733-7d6e-48d3-842f-0f469e69622b None None] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:53.235395 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74490733-7d6e-48d3-842f-0f469e69622b None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:53.235492 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74490733-7d6e-48d3-842f-0f469e69622b None None] PATH_INFO: `/` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:53.236625 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 426/857] 10.222.0.27 () {54 vars in 927 bytes} [Tue Aug 30 01:43:53 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 01:43:53.237298 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74490733-7d6e-48d3-842f-0f469e69622b None None] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:53.255924 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0649987-0506-4bdf-8a97-068fa792338a None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:53.290009 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e0de79f-b1da-4138-8116-f3f275f2527e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:53.290633 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e0de79f-b1da-4138-8116-f3f275f2527e None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:53.290911 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e0de79f-b1da-4138-8116-f3f275f2527e None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:53.291180 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e0de79f-b1da-4138-8116-f3f275f2527e None admin] PATH_INFO: `/v3/projects/service` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:53.297736 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e0de79f-b1da-4138-8116-f3f275f2527e None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:53.299172 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e0de79f-b1da-4138-8116-f3f275f2527e None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:53.310115 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application [None req-2e0de79f-b1da-4138-8116-f3f275f2527e None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 30 01:43:53.310115 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 01:43:53.310115 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 01:43:53.310115 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 01:43:53.310115 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 01:43:53.310115 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 01:43:53.310115 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 01:43:53.310115 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 01:43:53.310115 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 01:43:53.310115 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 01:43:53.310115 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 01:43:53.310115 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 01:43:53.310115 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 30 01:43:53.310115 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 30 01:43:53.310115 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 30 01:43:53.310996 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 30 01:43:53.310996 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 01:43:53.310996 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 01:43:53.310996 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 01:43:53.310996 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 01:43:53.310996 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 01:43:53.310996 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 01:43:53.310996 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 01:43:53.310996 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application with Lock( Aug 30 01:43:53.310996 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 01:43:53.310996 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._enter() Aug 30 01:43:53.310996 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 01:43:53.310996 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 01:43:53.310996 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 01:43:53.310996 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.creator() Aug 30 01:43:53.310996 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 01:43:53.310996 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application created_value = creator( Aug 30 01:43:53.311890 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 30 01:43:53.311890 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 30 01:43:53.311890 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 30 01:43:53.311890 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 30 01:43:53.311890 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 30 01:43:53.311890 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 30 01:43:53.311890 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 30 01:43:53.311890 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Aug 30 01:43:53.311890 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 432/858] 10.222.0.27 () {56 vars in 1111 bytes} [Tue Aug 30 01:43:53 2022] GET /identity/v3/projects/service => generated 88 bytes in 164 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 01:43:53.312376 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e0de79f-b1da-4138-8116-f3f275f2527e None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:53.341678 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74490733-7d6e-48d3-842f-0f469e69622b None None] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:53.361088 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:43:53.374698 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5a40b94-8477-48e3-bc79-a42ea80c2c48 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:53.375448 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5a40b94-8477-48e3-bc79-a42ea80c2c48 None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:53.375730 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5a40b94-8477-48e3-bc79-a42ea80c2c48 None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:53.375997 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5a40b94-8477-48e3-bc79-a42ea80c2c48 None admin] PATH_INFO: `/v3/services` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:53.376620 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5a40b94-8477-48e3-bc79-a42ea80c2c48 None admin] RBAC: Adding query filter params (name=volumev3) {{(pid=81578) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:43:53.376918 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5a40b94-8477-48e3-bc79-a42ea80c2c48 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:53.378297 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5a40b94-8477-48e3-bc79-a42ea80c2c48 None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:53.380615 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:43:53.383851 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 427/859] 10.222.0.27 () {56 vars in 1114 bytes} [Tue Aug 30 01:43:53 2022] GET /identity/v3/services?name=volumev3 => generated 125 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:43:53.385053 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5528e183-2fdc-41a1-8912-b5ad517839eb None None] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:53.385335 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5528e183-2fdc-41a1-8912-b5ad517839eb None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:53.385633 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5528e183-2fdc-41a1-8912-b5ad517839eb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:53.403658 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.auth.core [None req-5528e183-2fdc-41a1-8912-b5ad517839eb None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81578) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 01:43:53.404843 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-5528e183-2fdc-41a1-8912-b5ad517839eb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:43:53.412649 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e0de79f-b1da-4138-8116-f3f275f2527e None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:53.446516 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be68d940-43a9-49c4-b3e2-c4f61efb64dc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:53.447110 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be68d940-43a9-49c4-b3e2-c4f61efb64dc None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:53.447385 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be68d940-43a9-49c4-b3e2-c4f61efb64dc None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:53.447652 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be68d940-43a9-49c4-b3e2-c4f61efb64dc None admin] PATH_INFO: `/v3/domains` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:53.448275 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be68d940-43a9-49c4-b3e2-c4f61efb64dc None admin] RBAC: Adding query filter params (name=Default) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:43:53.448607 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be68d940-43a9-49c4-b3e2-c4f61efb64dc None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:53.450005 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be68d940-43a9-49c4-b3e2-c4f61efb64dc None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:53.461625 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 433/860] 10.222.0.27 () {56 vars in 1109 bytes} [Tue Aug 30 01:43:53 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:43:53.462409 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be68d940-43a9-49c4-b3e2-c4f61efb64dc None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:53.473139 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 428/861] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Aug 30 01:43:53 2022] POST /identity/v3/auth/tokens => generated 1161 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 30 01:43:53.474324 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5528e183-2fdc-41a1-8912-b5ad517839eb None None] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:53.578505 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be68d940-43a9-49c4-b3e2-c4f61efb64dc None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:53.583364 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5528e183-2fdc-41a1-8912-b5ad517839eb None None] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:53.588783 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:43:53.615140 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:43:53.615308 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf2c58bf-c3ff-4759-b48b-d384b32831cf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:53.615881 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf2c58bf-c3ff-4759-b48b-d384b32831cf None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:53.616156 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf2c58bf-c3ff-4759-b48b-d384b32831cf None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:53.616428 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf2c58bf-c3ff-4759-b48b-d384b32831cf None admin] PATH_INFO: `/v3/projects` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:53.617121 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf2c58bf-c3ff-4759-b48b-d384b32831cf None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:43:53.617424 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf2c58bf-c3ff-4759-b48b-d384b32831cf None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:53.618877 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf2c58bf-c3ff-4759-b48b-d384b32831cf None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:53.621674 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7991af30-8d25-43ea-9ce6-6f0d0ab9c8e4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:53.622463 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7991af30-8d25-43ea-9ce6-6f0d0ab9c8e4 None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:53.622913 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7991af30-8d25-43ea-9ce6-6f0d0ab9c8e4 None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:53.623366 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7991af30-8d25-43ea-9ce6-6f0d0ab9c8e4 None admin] PATH_INFO: `/v3/services` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:53.624165 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7991af30-8d25-43ea-9ce6-6f0d0ab9c8e4 None admin] RBAC: Adding query filter params (type=volumev3) {{(pid=81578) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:43:53.624637 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7991af30-8d25-43ea-9ce6-6f0d0ab9c8e4 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:53.626239 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7991af30-8d25-43ea-9ce6-6f0d0ab9c8e4 None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:53.631733 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 434/862] 10.222.0.27 () {56 vars in 1148 bytes} [Tue Aug 30 01:43:53 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:43:53.633493 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf2c58bf-c3ff-4759-b48b-d384b32831cf None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:53.642114 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 429/863] 10.222.0.27 () {56 vars in 1114 bytes} [Tue Aug 30 01:43:53 2022] GET /identity/v3/services?type=volumev3 => generated 125 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:43:53.643181 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7991af30-8d25-43ea-9ce6-6f0d0ab9c8e4 None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:53.774348 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf2c58bf-c3ff-4759-b48b-d384b32831cf None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:53.787654 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7991af30-8d25-43ea-9ce6-6f0d0ab9c8e4 None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:53.813630 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c88b17ab-a2b2-4d67-95d0-32ad966f5135 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:53.814260 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c88b17ab-a2b2-4d67-95d0-32ad966f5135 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:53.814540 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c88b17ab-a2b2-4d67-95d0-32ad966f5135 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:53.814835 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c88b17ab-a2b2-4d67-95d0-32ad966f5135 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:53.822662 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c88b17ab-a2b2-4d67-95d0-32ad966f5135 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:53.824105 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c88b17ab-a2b2-4d67-95d0-32ad966f5135 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:53.829946 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92d0e6e8-e702-4275-9df8-de9fb7b2ae42 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:53.830500 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92d0e6e8-e702-4275-9df8-de9fb7b2ae42 None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:53.830776 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92d0e6e8-e702-4275-9df8-de9fb7b2ae42 None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:53.831046 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92d0e6e8-e702-4275-9df8-de9fb7b2ae42 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:53.831767 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92d0e6e8-e702-4275-9df8-de9fb7b2ae42 None admin] RBAC: Adding query filter params (role.id=aa59b0c9009a480994d9df81a5f8d404, scope.project.id=4562f3951fbe465280073e60a11e44e2, user.id=121c374d49a64335ab74a431cfbc1320) {{(pid=81578) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:43:53.832077 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92d0e6e8-e702-4275-9df8-de9fb7b2ae42 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:53.834089 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92d0e6e8-e702-4275-9df8-de9fb7b2ae42 None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:53.836534 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application [None req-c88b17ab-a2b2-4d67-95d0-32ad966f5135 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 30 01:43:53.836534 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 01:43:53.836534 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 01:43:53.836534 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 01:43:53.836534 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 01:43:53.836534 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 01:43:53.836534 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 01:43:53.836534 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 01:43:53.836534 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 01:43:53.836534 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 01:43:53.836534 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 01:43:53.836534 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 01:43:53.836534 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 30 01:43:53.836534 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 30 01:43:53.836534 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 30 01:43:53.837407 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 30 01:43:53.837407 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 01:43:53.837407 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 01:43:53.837407 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 01:43:53.837407 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 01:43:53.837407 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 01:43:53.837407 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 01:43:53.837407 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 01:43:53.837407 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application with Lock( Aug 30 01:43:53.837407 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 01:43:53.837407 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._enter() Aug 30 01:43:53.837407 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 01:43:53.837407 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 01:43:53.837407 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 01:43:53.837407 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.creator() Aug 30 01:43:53.837407 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 01:43:53.837407 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application created_value = creator( Aug 30 01:43:53.838335 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 30 01:43:53.838335 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 30 01:43:53.838335 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 30 01:43:53.838335 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 30 01:43:53.838335 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 30 01:43:53.838335 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 30 01:43:53.838335 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 30 01:43:53.838335 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Aug 30 01:43:53.838335 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 435/864] 10.222.0.27 () {56 vars in 1111 bytes} [Tue Aug 30 01:43:53 2022] GET /identity/v3/projects/service => generated 88 bytes in 205 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 01:43:53.839856 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c88b17ab-a2b2-4d67-95d0-32ad966f5135 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:53.850326 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 430/865] 10.222.0.27 () {56 vars in 1374 bytes} [Tue Aug 30 01:43:53 2022] GET /identity/v3/role_assignments?user.id=121c374d49a64335ab74a431cfbc1320&scope.project.id=4562f3951fbe465280073e60a11e44e2&role.id=aa59b0c9009a480994d9df81a5f8d404 => generated 608 bytes in 203 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:43:53.850326 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92d0e6e8-e702-4275-9df8-de9fb7b2ae42 None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:53.877869 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:43:53.926714 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:43:53.979745 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92d0e6e8-e702-4275-9df8-de9fb7b2ae42 None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:53.990955 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c88b17ab-a2b2-4d67-95d0-32ad966f5135 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:54.024874 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fb5bd90-029f-4440-a85a-3eb14af69460 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:54.025562 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fb5bd90-029f-4440-a85a-3eb14af69460 None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:54.025791 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fb5bd90-029f-4440-a85a-3eb14af69460 None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:54.026063 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fb5bd90-029f-4440-a85a-3eb14af69460 None admin] PATH_INFO: `/v3/projects` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:54.027500 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2fb5bd90-029f-4440-a85a-3eb14af69460 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81578) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:43:54.027500 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2fb5bd90-029f-4440-a85a-3eb14af69460 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:54.028470 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2fb5bd90-029f-4440-a85a-3eb14af69460 None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:54.033908 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b228c325-ff50-47dd-aacb-00ce4459e34d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:54.034835 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b228c325-ff50-47dd-aacb-00ce4459e34d None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:54.035319 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b228c325-ff50-47dd-aacb-00ce4459e34d None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:54.036967 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b228c325-ff50-47dd-aacb-00ce4459e34d None admin] PATH_INFO: `/v3/services/compute_legacy` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:54.037807 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b228c325-ff50-47dd-aacb-00ce4459e34d None admin] RBAC: Authorizing `identity:get_service(service_id=compute_legacy)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:54.040265 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b228c325-ff50-47dd-aacb-00ce4459e34d None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:54.042062 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 431/866] 10.222.0.27 () {56 vars in 1148 bytes} [Tue Aug 30 01:43:53 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 196 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:43:54.048489 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fb5bd90-029f-4440-a85a-3eb14af69460 None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:54.057008 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application [None req-b228c325-ff50-47dd-aacb-00ce4459e34d None admin] Could not find service: compute_legacy.: keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Aug 30 01:43:54.057008 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 01:43:54.057008 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Aug 30 01:43:54.057008 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Aug 30 01:43:54.057008 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Aug 30 01:43:54.057008 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Aug 30 01:43:54.057008 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Aug 30 01:43:54.057008 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 30 01:43:54.057008 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Aug 30 01:43:54.057008 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Aug 30 01:43:54.057008 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 30 01:43:54.057008 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Aug 30 01:43:54.057008 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 01:43:54.057008 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 01:43:54.057008 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 01:43:54.057008 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 01:43:54.058040 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 01:43:54.058040 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 01:43:54.058040 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 01:43:54.058040 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 01:43:54.058040 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 01:43:54.058040 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 01:43:54.058040 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 01:43:54.058040 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Aug 30 01:43:54.058040 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_service(service_id) Aug 30 01:43:54.058040 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Aug 30 01:43:54.058040 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Aug 30 01:43:54.058040 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 01:43:54.058040 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 01:43:54.058040 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 01:43:54.058040 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 01:43:54.058040 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 01:43:54.058040 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 01:43:54.058946 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 01:43:54.058946 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application with Lock( Aug 30 01:43:54.058946 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 01:43:54.058946 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._enter() Aug 30 01:43:54.058946 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 01:43:54.058946 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 01:43:54.058946 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 01:43:54.058946 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.creator() Aug 30 01:43:54.058946 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 01:43:54.058946 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application created_value = creator( Aug 30 01:43:54.058946 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Aug 30 01:43:54.058946 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 30 01:43:54.058946 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Aug 30 01:43:54.058946 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Aug 30 01:43:54.058946 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 436/867] 10.222.0.27 () {56 vars in 1132 bytes} [Tue Aug 30 01:43:53 2022] GET /identity/v3/services/compute_legacy => generated 95 bytes in 220 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 01:43:54.067144 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b228c325-ff50-47dd-aacb-00ce4459e34d None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:54.168139 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:43:54.169891 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fb5bd90-029f-4440-a85a-3eb14af69460 None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:54.177077 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b228c325-ff50-47dd-aacb-00ce4459e34d None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:54.188496 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:43:54.206002 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba45756d-8911-4319-bb21-33c06f849c1e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:54.206597 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba45756d-8911-4319-bb21-33c06f849c1e None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:54.206872 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba45756d-8911-4319-bb21-33c06f849c1e None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:54.207133 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba45756d-8911-4319-bb21-33c06f849c1e None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:54.207914 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba45756d-8911-4319-bb21-33c06f849c1e None admin] RBAC: Adding query filter params (role.id=ead5fac86a0747ea8823a82efcea8ed7, scope.project.id=4562f3951fbe465280073e60a11e44e2, user.id=b4f0008fde424f22bc1f7307cf422774) {{(pid=81578) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:43:54.208848 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba45756d-8911-4319-bb21-33c06f849c1e None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:54.210437 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba45756d-8911-4319-bb21-33c06f849c1e None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:54.212681 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-736a3b23-ea56-46e5-9513-17dace9de5e7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:54.213285 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-736a3b23-ea56-46e5-9513-17dace9de5e7 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:54.213593 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-736a3b23-ea56-46e5-9513-17dace9de5e7 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:54.213867 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-736a3b23-ea56-46e5-9513-17dace9de5e7 None admin] PATH_INFO: `/v3/services` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:54.214492 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-736a3b23-ea56-46e5-9513-17dace9de5e7 None admin] RBAC: Adding query filter params (name=compute_legacy) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:43:54.214789 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-736a3b23-ea56-46e5-9513-17dace9de5e7 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:54.216140 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-736a3b23-ea56-46e5-9513-17dace9de5e7 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:54.216700 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 432/868] 10.222.0.27 () {56 vars in 1374 bytes} [Tue Aug 30 01:43:54 2022] GET /identity/v3/role_assignments?user.id=b4f0008fde424f22bc1f7307cf422774&scope.project.id=4562f3951fbe465280073e60a11e44e2&role.id=ead5fac86a0747ea8823a82efcea8ed7 => generated 608 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:43:54.229731 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 437/869] 10.222.0.27 () {56 vars in 1126 bytes} [Tue Aug 30 01:43:54 2022] GET /identity/v3/services?name=compute_legacy => generated 131 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:43:54.238745 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba45756d-8911-4319-bb21-33c06f849c1e None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:54.361471 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba45756d-8911-4319-bb21-33c06f849c1e None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:54.398368 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a5695e7-35c1-48d8-979c-5e076341f981 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:54.399069 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a5695e7-35c1-48d8-979c-5e076341f981 None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:54.399410 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a5695e7-35c1-48d8-979c-5e076341f981 None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:54.399743 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a5695e7-35c1-48d8-979c-5e076341f981 None admin] PATH_INFO: `/v3/services` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:54.400441 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a5695e7-35c1-48d8-979c-5e076341f981 None admin] RBAC: Adding query filter params (type=compute_legacy) {{(pid=81578) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:43:54.400810 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a5695e7-35c1-48d8-979c-5e076341f981 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:54.402264 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a5695e7-35c1-48d8-979c-5e076341f981 None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:54.407245 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 433/870] 10.222.0.27 () {56 vars in 1126 bytes} [Tue Aug 30 01:43:54 2022] GET /identity/v3/services?type=compute_legacy => generated 131 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:43:55.623222 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfd3b73a-0719-46d6-a189-536e32ee4c96 None None] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:55.623774 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfd3b73a-0719-46d6-a189-536e32ee4c96 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:55.623774 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfd3b73a-0719-46d6-a189-536e32ee4c96 None None] PATH_INFO: `/` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:55.625355 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 438/871] 10.222.0.27 () {54 vars in 927 bytes} [Tue Aug 30 01:43:55 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 01:43:55.631007 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae5f9204-0777-4f89-ab26-854e25e8895b None None] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:55.631421 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae5f9204-0777-4f89-ab26-854e25e8895b None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:55.631607 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae5f9204-0777-4f89-ab26-854e25e8895b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:55.651092 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.auth.core [None req-ae5f9204-0777-4f89-ab26-854e25e8895b None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81578) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 01:43:55.652349 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-ae5f9204-0777-4f89-ab26-854e25e8895b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:43:55.698158 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:43:55.728690 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 434/872] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Aug 30 01:43:55 2022] POST /identity/v3/auth/tokens => generated 1161 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 30 01:43:55.912294 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dea63262-44ee-4903-9df6-8202f803a408 None None] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:55.912538 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dea63262-44ee-4903-9df6-8202f803a408 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:55.912772 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dea63262-44ee-4903-9df6-8202f803a408 None None] PATH_INFO: `/` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:55.914383 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 439/873] 10.222.0.27 () {54 vars in 927 bytes} [Tue Aug 30 01:43:55 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 01:43:55.919674 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-054ced44-3750-4a66-b1d1-130e21b4e17d None None] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:55.920197 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-054ced44-3750-4a66-b1d1-130e21b4e17d None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:55.920677 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-054ced44-3750-4a66-b1d1-130e21b4e17d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:55.940500 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.auth.core [None req-054ced44-3750-4a66-b1d1-130e21b4e17d None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81578) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 01:43:55.942083 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-054ced44-3750-4a66-b1d1-130e21b4e17d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:43:56.011920 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 435/874] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Aug 30 01:43:55 2022] POST /identity/v3/auth/tokens => generated 1161 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 30 01:43:56.021425 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dea63262-44ee-4903-9df6-8202f803a408 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:56.132381 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dea63262-44ee-4903-9df6-8202f803a408 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:56.168482 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b434d1d-314f-440b-99bc-e58c5bbd38e9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:56.169015 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b434d1d-314f-440b-99bc-e58c5bbd38e9 None admin] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:56.169287 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b434d1d-314f-440b-99bc-e58c5bbd38e9 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:56.169591 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b434d1d-314f-440b-99bc-e58c5bbd38e9 None admin] PATH_INFO: `/v3/services` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:56.170255 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b434d1d-314f-440b-99bc-e58c5bbd38e9 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:56.171672 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b434d1d-314f-440b-99bc-e58c5bbd38e9 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:56.248870 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 440/875] 10.222.0.27 () {60 vars in 1141 bytes} [Tue Aug 30 01:43:56 2022] POST /identity/v3/services => generated 251 bytes in 228 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 01:43:56.262269 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6baf0671-4ce9-47ae-abfe-ea2bee2deec3 None None] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:56.262901 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6baf0671-4ce9-47ae-abfe-ea2bee2deec3 None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:56.263379 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6baf0671-4ce9-47ae-abfe-ea2bee2deec3 None None] PATH_INFO: `/` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:56.264996 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 436/876] 10.222.0.27 () {54 vars in 927 bytes} [Tue Aug 30 01:43:56 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 01:43:56.273503 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbb174c6-8544-46bd-89ca-fda93a3a1f01 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:56.273799 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbb174c6-8544-46bd-89ca-fda93a3a1f01 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:56.274023 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbb174c6-8544-46bd-89ca-fda93a3a1f01 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:56.300847 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-fbb174c6-8544-46bd-89ca-fda93a3a1f01 None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 01:43:56.302751 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-fbb174c6-8544-46bd-89ca-fda93a3a1f01 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:43:56.303585 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6baf0671-4ce9-47ae-abfe-ea2bee2deec3 None None] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:56.320723 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:43:56.397504 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 441/877] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Aug 30 01:43:56 2022] POST /identity/v3/auth/tokens => generated 1262 bytes in 121 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 30 01:43:56.397504 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e2f302f-304f-444c-8f7e-332cda878132 None None] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:56.397504 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e2f302f-304f-444c-8f7e-332cda878132 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:56.397504 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e2f302f-304f-444c-8f7e-332cda878132 None None] PATH_INFO: `/` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:56.397504 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 442/878] 10.222.0.27 () {54 vars in 927 bytes} [Tue Aug 30 01:43:56 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 01:43:56.402426 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3a6c1af-38fb-467a-b704-59c8bf6169ba None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:56.402707 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3a6c1af-38fb-467a-b704-59c8bf6169ba None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:56.402974 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3a6c1af-38fb-467a-b704-59c8bf6169ba None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:56.424176 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-f3a6c1af-38fb-467a-b704-59c8bf6169ba None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 01:43:56.425839 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-f3a6c1af-38fb-467a-b704-59c8bf6169ba None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:43:56.435005 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6baf0671-4ce9-47ae-abfe-ea2bee2deec3 None None] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:56.444087 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:43:56.479509 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65f2b3e4-3dcb-4332-ab28-3612af5dda6b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:56.480153 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65f2b3e4-3dcb-4332-ab28-3612af5dda6b None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:56.480750 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65f2b3e4-3dcb-4332-ab28-3612af5dda6b None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:56.481027 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65f2b3e4-3dcb-4332-ab28-3612af5dda6b None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:56.497605 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65f2b3e4-3dcb-4332-ab28-3612af5dda6b None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 30 01:43:56.507022 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application [None req-65f2b3e4-3dcb-4332-ab28-3612af5dda6b None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Aug 30 01:43:56.507022 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 01:43:56.507022 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 01:43:56.507022 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 01:43:56.507022 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 01:43:56.507022 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 01:43:56.507022 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 01:43:56.507022 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 01:43:56.507022 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 01:43:56.507022 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 01:43:56.507022 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 01:43:56.507022 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 01:43:56.507022 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 30 01:43:56.507022 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 30 01:43:56.507022 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 30 01:43:56.507022 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 30 01:43:56.508842 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 30 01:43:56.508842 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 30 01:43:56.508842 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 30 01:43:56.508842 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 30 01:43:56.508842 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 01:43:56.508842 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 01:43:56.508842 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 01:43:56.508842 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 01:43:56.508842 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 01:43:56.508842 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 01:43:56.508842 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 01:43:56.508842 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application with Lock( Aug 30 01:43:56.508842 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 01:43:56.508842 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._enter() Aug 30 01:43:56.508842 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 01:43:56.508842 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 01:43:56.510482 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 01:43:56.510482 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.creator() Aug 30 01:43:56.510482 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 01:43:56.510482 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application created_value = creator( Aug 30 01:43:56.510482 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 30 01:43:56.510482 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 30 01:43:56.510482 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 30 01:43:56.510482 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 30 01:43:56.510482 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 30 01:43:56.510482 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 30 01:43:56.510482 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Aug 30 01:43:56.510482 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application Aug 30 01:43:56.510482 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 437/879] 10.222.0.27 () {56 vars in 1120 bytes} [Tue Aug 30 01:43:56 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 206 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 30 01:43:56.514593 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65f2b3e4-3dcb-4332-ab28-3612af5dda6b None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:56.523060 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 443/880] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Aug 30 01:43:56 2022] POST /identity/v3/auth/tokens => generated 1262 bytes in 121 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 30 01:43:56.625983 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65f2b3e4-3dcb-4332-ab28-3612af5dda6b None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:56.660518 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9995f30d-c7fa-4e32-9139-fff4b958d265 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:56.661725 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9995f30d-c7fa-4e32-9139-fff4b958d265 None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:56.661998 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9995f30d-c7fa-4e32-9139-fff4b958d265 None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:56.662308 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9995f30d-c7fa-4e32-9139-fff4b958d265 None admin] PATH_INFO: `/v3/roles` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:56.662975 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9995f30d-c7fa-4e32-9139-fff4b958d265 None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=81578) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:43:56.663273 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9995f30d-c7fa-4e32-9139-fff4b958d265 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:56.664615 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9995f30d-c7fa-4e32-9139-fff4b958d265 None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:56.670026 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3a6c1af-38fb-467a-b704-59c8bf6169ba None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:56.674306 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 438/881] 10.222.0.27 () {56 vars in 1115 bytes} [Tue Aug 30 01:43:56 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 30 01:43:56.678957 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9995f30d-c7fa-4e32-9139-fff4b958d265 None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:56.713084 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:43:56.773436 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:43:56.811782 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3a6c1af-38fb-467a-b704-59c8bf6169ba None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:56.812649 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9995f30d-c7fa-4e32-9139-fff4b958d265 None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:56.853022 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19ad9257-de0e-47b2-ac38-fcb7ce687448 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:56.853231 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30f6da66-8c85-4ee2-960c-1e693e2c9558 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:56.853678 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19ad9257-de0e-47b2-ac38-fcb7ce687448 None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:56.853784 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30f6da66-8c85-4ee2-960c-1e693e2c9558 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:56.853958 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19ad9257-de0e-47b2-ac38-fcb7ce687448 None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:56.854055 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30f6da66-8c85-4ee2-960c-1e693e2c9558 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:56.854222 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19ad9257-de0e-47b2-ac38-fcb7ce687448 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:56.854319 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30f6da66-8c85-4ee2-960c-1e693e2c9558 None admin] PATH_INFO: `/v3/services/object-store` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:56.854840 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30f6da66-8c85-4ee2-960c-1e693e2c9558 None admin] RBAC: Authorizing `identity:get_service(service_id=object-store)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:56.856208 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30f6da66-8c85-4ee2-960c-1e693e2c9558 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:56.865812 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application [None req-30f6da66-8c85-4ee2-960c-1e693e2c9558 None admin] Could not find service: object-store.: keystone.exception.ServiceNotFound: Could not find service: object-store. Aug 30 01:43:56.865812 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 01:43:56.865812 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Aug 30 01:43:56.865812 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Aug 30 01:43:56.865812 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Aug 30 01:43:56.865812 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Aug 30 01:43:56.865812 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Aug 30 01:43:56.865812 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 30 01:43:56.865812 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Aug 30 01:43:56.865812 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Aug 30 01:43:56.865812 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 30 01:43:56.865812 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Aug 30 01:43:56.865812 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 01:43:56.865812 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 01:43:56.865812 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 01:43:56.865812 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 01:43:56.865812 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 01:43:56.866825 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 01:43:56.866825 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 01:43:56.866825 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 01:43:56.866825 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 01:43:56.866825 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 01:43:56.866825 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 01:43:56.866825 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Aug 30 01:43:56.866825 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_service(service_id) Aug 30 01:43:56.866825 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Aug 30 01:43:56.866825 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Aug 30 01:43:56.866825 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 01:43:56.866825 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 01:43:56.866825 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 01:43:56.866825 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 01:43:56.866825 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 01:43:56.866825 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 01:43:56.870616 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 01:43:56.870616 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application with Lock( Aug 30 01:43:56.870616 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 01:43:56.870616 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._enter() Aug 30 01:43:56.870616 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 01:43:56.870616 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 01:43:56.870616 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 01:43:56.870616 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.creator() Aug 30 01:43:56.870616 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 01:43:56.870616 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application created_value = creator( Aug 30 01:43:56.870616 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Aug 30 01:43:56.870616 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 30 01:43:56.870616 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Aug 30 01:43:56.870616 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Aug 30 01:43:56.871376 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19ad9257-de0e-47b2-ac38-fcb7ce687448 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:56.871679 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 444/882] 10.222.0.27 () {56 vars in 1126 bytes} [Tue Aug 30 01:43:56 2022] GET /identity/v3/services/object-store => generated 93 bytes in 198 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 30 01:43:56.871757 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19ad9257-de0e-47b2-ac38-fcb7ce687448 None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:56.871826 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30f6da66-8c85-4ee2-960c-1e693e2c9558 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:56.886171 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application [None req-19ad9257-de0e-47b2-ac38-fcb7ce687448 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 01:43:56.886171 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 01:43:56.886171 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 01:43:56.886171 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 01:43:56.886171 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 01:43:56.886171 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 01:43:56.886171 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 01:43:56.886171 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 01:43:56.886171 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 01:43:56.886171 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 01:43:56.886171 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 01:43:56.886171 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 01:43:56.886171 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 30 01:43:56.886171 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 30 01:43:56.886171 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 30 01:43:56.886171 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 30 01:43:56.887330 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 01:43:56.887330 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 01:43:56.887330 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 01:43:56.887330 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 01:43:56.887330 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 01:43:56.887330 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 01:43:56.887330 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 01:43:56.887330 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application with Lock( Aug 30 01:43:56.887330 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 01:43:56.887330 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._enter() Aug 30 01:43:56.887330 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 01:43:56.887330 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 01:43:56.887330 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 01:43:56.887330 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.creator() Aug 30 01:43:56.887330 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 01:43:56.887330 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application created_value = creator( Aug 30 01:43:56.887330 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 30 01:43:56.888396 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 30 01:43:56.888396 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 01:43:56.888396 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application Aug 30 01:43:56.888396 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 439/883] 10.222.0.27 () {56 vars in 1108 bytes} [Tue Aug 30 01:43:56 2022] GET /identity/v3/domains/Default => generated 87 bytes in 209 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 30 01:43:56.888396 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19ad9257-de0e-47b2-ac38-fcb7ce687448 None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:56.975245 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:43:56.988352 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30f6da66-8c85-4ee2-960c-1e693e2c9558 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:56.994565 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19ad9257-de0e-47b2-ac38-fcb7ce687448 None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:57.024093 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac30c30c-bc01-4ca6-b1cd-86c41d70438c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:57.024689 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac30c30c-bc01-4ca6-b1cd-86c41d70438c None admin] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:57.024968 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac30c30c-bc01-4ca6-b1cd-86c41d70438c None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:57.025235 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac30c30c-bc01-4ca6-b1cd-86c41d70438c None admin] PATH_INFO: `/v3/services` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:57.025914 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac30c30c-bc01-4ca6-b1cd-86c41d70438c None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:57.027269 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac30c30c-bc01-4ca6-b1cd-86c41d70438c None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:57.030697 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-936e14c3-9945-464f-9f07-0bdba78ed780 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:57.031305 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-936e14c3-9945-464f-9f07-0bdba78ed780 None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:57.031438 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:43:57.031580 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-936e14c3-9945-464f-9f07-0bdba78ed780 None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:57.032141 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-936e14c3-9945-464f-9f07-0bdba78ed780 None admin] PATH_INFO: `/v3/services` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:57.032468 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-936e14c3-9945-464f-9f07-0bdba78ed780 None admin] RBAC: Adding query filter params (name=object-store) {{(pid=81578) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:43:57.032810 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-936e14c3-9945-464f-9f07-0bdba78ed780 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:57.034143 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-936e14c3-9945-464f-9f07-0bdba78ed780 None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:57.039368 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 440/884] 10.222.0.27 () {56 vars in 1122 bytes} [Tue Aug 30 01:43:56 2022] GET /identity/v3/services?name=object-store => generated 129 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:43:57.040143 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-936e14c3-9945-464f-9f07-0bdba78ed780 None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:57.116194 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 445/885] 10.222.0.27 () {60 vars in 1141 bytes} [Tue Aug 30 01:43:56 2022] POST /identity/v3/services => generated 269 bytes in 248 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 01:43:57.125660 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac30c30c-bc01-4ca6-b1cd-86c41d70438c None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:57.155123 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-936e14c3-9945-464f-9f07-0bdba78ed780 None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:57.201081 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34dca794-9959-46fd-921c-2858e3be3490 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:57.201713 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34dca794-9959-46fd-921c-2858e3be3490 None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:57.202140 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34dca794-9959-46fd-921c-2858e3be3490 None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:57.202461 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34dca794-9959-46fd-921c-2858e3be3490 None admin] PATH_INFO: `/v3/domains` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:57.203107 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34dca794-9959-46fd-921c-2858e3be3490 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81578) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:43:57.203408 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34dca794-9959-46fd-921c-2858e3be3490 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:57.204838 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34dca794-9959-46fd-921c-2858e3be3490 None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:57.215028 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:43:57.219711 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 441/886] 10.222.0.27 () {56 vars in 1109 bytes} [Tue Aug 30 01:43:57 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 180 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:43:57.225200 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34dca794-9959-46fd-921c-2858e3be3490 None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:57.253996 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac30c30c-bc01-4ca6-b1cd-86c41d70438c None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:57.293442 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61e4b750-4113-4249-b2e4-f40d391613cb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:57.294437 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61e4b750-4113-4249-b2e4-f40d391613cb None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:57.294922 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61e4b750-4113-4249-b2e4-f40d391613cb None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:57.295564 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61e4b750-4113-4249-b2e4-f40d391613cb None admin] PATH_INFO: `/v3/services` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:57.296510 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61e4b750-4113-4249-b2e4-f40d391613cb None admin] RBAC: Adding query filter params (type=object-store) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:43:57.297017 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61e4b750-4113-4249-b2e4-f40d391613cb None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:57.298618 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61e4b750-4113-4249-b2e4-f40d391613cb None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:57.304422 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 446/887] 10.222.0.27 () {56 vars in 1122 bytes} [Tue Aug 30 01:43:57 2022] GET /identity/v3/services?type=object-store => generated 129 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:43:57.351994 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34dca794-9959-46fd-921c-2858e3be3490 None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:57.394315 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-556f532a-2673-4e6d-b3d6-5ef01c756498 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:57.395908 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-556f532a-2673-4e6d-b3d6-5ef01c756498 None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:57.396436 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-556f532a-2673-4e6d-b3d6-5ef01c756498 None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:57.396932 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-556f532a-2673-4e6d-b3d6-5ef01c756498 None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:57.405771 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-556f532a-2673-4e6d-b3d6-5ef01c756498 None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:57.407570 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-556f532a-2673-4e6d-b3d6-5ef01c756498 None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:57.421319 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application [None req-556f532a-2673-4e6d-b3d6-5ef01c756498 None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Aug 30 01:43:57.421319 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 01:43:57.421319 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 01:43:57.421319 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 01:43:57.421319 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 01:43:57.421319 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 01:43:57.421319 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 01:43:57.421319 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 01:43:57.421319 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 01:43:57.421319 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 01:43:57.421319 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 01:43:57.421319 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 01:43:57.421319 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 30 01:43:57.421319 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 30 01:43:57.421319 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 30 01:43:57.421319 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 30 01:43:57.422391 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 01:43:57.422391 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 01:43:57.422391 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 30 01:43:57.422391 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 30 01:43:57.422391 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 30 01:43:57.422391 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 30 01:43:57.422391 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 01:43:57.422391 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 01:43:57.422391 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 01:43:57.422391 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 01:43:57.422391 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 01:43:57.422391 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application with Lock( Aug 30 01:43:57.422391 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 01:43:57.422391 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._enter() Aug 30 01:43:57.422391 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 01:43:57.422391 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 01:43:57.422391 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 01:43:57.422391 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.creator() Aug 30 01:43:57.423334 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 01:43:57.423334 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application created_value = creator( Aug 30 01:43:57.423334 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 30 01:43:57.423334 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 30 01:43:57.423334 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 30 01:43:57.423334 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 30 01:43:57.423334 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 30 01:43:57.423334 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 30 01:43:57.423334 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Aug 30 01:43:57.423334 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application Aug 30 01:43:57.426011 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 442/888] 10.222.0.27 () {56 vars in 1117 bytes} [Tue Aug 30 01:43:57 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 201 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 30 01:43:57.432565 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61e4b750-4113-4249-b2e4-f40d391613cb None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:57.500248 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:43:57.543655 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61e4b750-4113-4249-b2e4-f40d391613cb None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:57.578520 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b67c526f-9894-4e19-af76-6a84a2c19754 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:57.579192 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b67c526f-9894-4e19-af76-6a84a2c19754 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:57.579471 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b67c526f-9894-4e19-af76-6a84a2c19754 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:57.579743 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b67c526f-9894-4e19-af76-6a84a2c19754 None admin] PATH_INFO: `/v3/users` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:57.580522 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b67c526f-9894-4e19-af76-6a84a2c19754 None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:43:57.580833 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b67c526f-9894-4e19-af76-6a84a2c19754 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:57.582413 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b67c526f-9894-4e19-af76-6a84a2c19754 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:57.605570 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 447/889] 10.222.0.27 () {56 vars in 1149 bytes} [Tue Aug 30 01:43:57 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:43:57.607163 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-556f532a-2673-4e6d-b3d6-5ef01c756498 None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:57.675675 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:43:57.720540 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-556f532a-2673-4e6d-b3d6-5ef01c756498 None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:57.757198 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ae445f4-8c78-4fc0-97c5-18967d4e14d3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:57.757834 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ae445f4-8c78-4fc0-97c5-18967d4e14d3 None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:57.758115 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ae445f4-8c78-4fc0-97c5-18967d4e14d3 None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:57.758385 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ae445f4-8c78-4fc0-97c5-18967d4e14d3 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:57.829281 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ae445f4-8c78-4fc0-97c5-18967d4e14d3 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:57.830836 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ae445f4-8c78-4fc0-97c5-18967d4e14d3 None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:57.893802 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:43:57.935192 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 443/890] 10.222.0.27 () {58 vars in 1282 bytes} [Tue Aug 30 01:43:57 2022] GET /identity/v3/auth/tokens => generated 1372 bytes in 328 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Aug 30 01:43:57.940838 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b67c526f-9894-4e19-af76-6a84a2c19754 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:58.055082 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b67c526f-9894-4e19-af76-6a84a2c19754 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:58.073385 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:43:58.091342 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29d8d135-24ba-4e98-9fa5-d27e3f25c203 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:58.091912 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29d8d135-24ba-4e98-9fa5-d27e3f25c203 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:58.092194 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29d8d135-24ba-4e98-9fa5-d27e3f25c203 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:58.092464 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29d8d135-24ba-4e98-9fa5-d27e3f25c203 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:58.104113 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29d8d135-24ba-4e98-9fa5-d27e3f25c203 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:58.105660 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29d8d135-24ba-4e98-9fa5-d27e3f25c203 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:58.120952 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application [None req-29d8d135-24ba-4e98-9fa5-d27e3f25c203 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 01:43:58.120952 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 01:43:58.120952 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 01:43:58.120952 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 01:43:58.120952 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 01:43:58.120952 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 01:43:58.120952 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 01:43:58.120952 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 01:43:58.120952 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 01:43:58.120952 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 01:43:58.120952 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 01:43:58.120952 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 01:43:58.120952 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 30 01:43:58.120952 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 30 01:43:58.120952 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 30 01:43:58.120952 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 30 01:43:58.121934 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 01:43:58.121934 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 01:43:58.121934 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 01:43:58.121934 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 01:43:58.121934 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 01:43:58.121934 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 01:43:58.121934 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 01:43:58.121934 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application with Lock( Aug 30 01:43:58.121934 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 01:43:58.121934 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._enter() Aug 30 01:43:58.121934 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 01:43:58.121934 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 01:43:58.121934 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 01:43:58.121934 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.creator() Aug 30 01:43:58.121934 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 01:43:58.121934 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application created_value = creator( Aug 30 01:43:58.121934 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 30 01:43:58.123220 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 30 01:43:58.123220 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 01:43:58.123220 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Aug 30 01:43:58.123220 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 448/891] 10.222.0.27 () {56 vars in 1108 bytes} [Tue Aug 30 01:43:57 2022] GET /identity/v3/domains/Default => generated 87 bytes in 182 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 01:43:58.128412 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ae445f4-8c78-4fc0-97c5-18967d4e14d3 None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:58.232996 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ae445f4-8c78-4fc0-97c5-18967d4e14d3 None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:58.268136 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bab10d9-a4dc-4975-8f77-a91892079421 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:58.268810 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bab10d9-a4dc-4975-8f77-a91892079421 None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:58.269337 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bab10d9-a4dc-4975-8f77-a91892079421 None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:58.269642 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bab10d9-a4dc-4975-8f77-a91892079421 None admin] PATH_INFO: `/v3/domains` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:58.271435 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0bab10d9-a4dc-4975-8f77-a91892079421 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81578) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:43:58.271435 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0bab10d9-a4dc-4975-8f77-a91892079421 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:58.272552 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0bab10d9-a4dc-4975-8f77-a91892079421 None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:58.284925 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 444/892] 10.222.0.27 () {56 vars in 1109 bytes} [Tue Aug 30 01:43:58 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:43:58.291748 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29d8d135-24ba-4e98-9fa5-d27e3f25c203 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:58.356917 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a25cdd7-73d6-4435-8556-6346ae7ddfbf None None] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:58.357216 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a25cdd7-73d6-4435-8556-6346ae7ddfbf None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:58.357492 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a25cdd7-73d6-4435-8556-6346ae7ddfbf None None] PATH_INFO: `/` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:58.358944 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 445/893] 10.222.0.27 () {54 vars in 927 bytes} [Tue Aug 30 01:43:58 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 01:43:58.364075 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b79ffdc-2527-4d3a-a581-3da6c426db67 None None] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:58.364876 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b79ffdc-2527-4d3a-a581-3da6c426db67 None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:58.365278 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b79ffdc-2527-4d3a-a581-3da6c426db67 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:58.384864 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.auth.core [None req-3b79ffdc-2527-4d3a-a581-3da6c426db67 None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81578) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 01:43:58.386088 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-3b79ffdc-2527-4d3a-a581-3da6c426db67 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:43:58.402920 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:43:58.416198 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29d8d135-24ba-4e98-9fa5-d27e3f25c203 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:58.453390 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acf95c20-2f78-4fe8-9ab7-974b33dd4ad8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:58.454480 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acf95c20-2f78-4fe8-9ab7-974b33dd4ad8 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:58.454861 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acf95c20-2f78-4fe8-9ab7-974b33dd4ad8 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:58.455293 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acf95c20-2f78-4fe8-9ab7-974b33dd4ad8 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:58.461730 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 446/894] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Aug 30 01:43:58 2022] POST /identity/v3/auth/tokens => generated 1372 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Aug 30 01:43:58.463231 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-acf95c20-2f78-4fe8-9ab7-974b33dd4ad8 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:58.464781 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-acf95c20-2f78-4fe8-9ab7-974b33dd4ad8 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:58.477665 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application [None req-acf95c20-2f78-4fe8-9ab7-974b33dd4ad8 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 30 01:43:58.477665 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 01:43:58.477665 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 01:43:58.477665 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 01:43:58.477665 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 01:43:58.477665 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 01:43:58.477665 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 01:43:58.477665 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 01:43:58.477665 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 01:43:58.477665 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 01:43:58.477665 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 01:43:58.477665 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 01:43:58.477665 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 30 01:43:58.477665 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 30 01:43:58.477665 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 30 01:43:58.478579 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 30 01:43:58.478579 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 01:43:58.478579 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 01:43:58.478579 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 01:43:58.478579 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 01:43:58.478579 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 01:43:58.478579 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 01:43:58.478579 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 01:43:58.478579 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application with Lock( Aug 30 01:43:58.478579 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 01:43:58.478579 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._enter() Aug 30 01:43:58.478579 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 01:43:58.478579 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 01:43:58.478579 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 01:43:58.478579 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.creator() Aug 30 01:43:58.478579 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 01:43:58.478579 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application created_value = creator( Aug 30 01:43:58.479619 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 30 01:43:58.479619 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 30 01:43:58.479619 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 30 01:43:58.479619 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 30 01:43:58.479619 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 30 01:43:58.479619 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 30 01:43:58.479619 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 30 01:43:58.479619 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Aug 30 01:43:58.479619 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 449/895] 10.222.0.27 () {56 vars in 1111 bytes} [Tue Aug 30 01:43:58 2022] GET /identity/v3/projects/service => generated 88 bytes in 188 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 30 01:43:58.484675 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b79ffdc-2527-4d3a-a581-3da6c426db67 None None] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:58.591451 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b79ffdc-2527-4d3a-a581-3da6c426db67 None None] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:58.628525 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bfc85d9-529b-4a65-87ba-ab96879be1af None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:58.629051 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bfc85d9-529b-4a65-87ba-ab96879be1af None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:58.629330 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bfc85d9-529b-4a65-87ba-ab96879be1af None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:58.629687 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bfc85d9-529b-4a65-87ba-ab96879be1af None admin] PATH_INFO: `/v3/projects` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:58.630644 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0bfc85d9-529b-4a65-87ba-ab96879be1af None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81578) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:43:58.630954 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0bfc85d9-529b-4a65-87ba-ab96879be1af None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:58.632373 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0bfc85d9-529b-4a65-87ba-ab96879be1af None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:58.644551 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 447/896] 10.222.0.27 () {56 vars in 1148 bytes} [Tue Aug 30 01:43:58 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:43:58.650548 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acf95c20-2f78-4fe8-9ab7-974b33dd4ad8 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:58.694039 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:43:58.766470 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bfc85d9-529b-4a65-87ba-ab96879be1af None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:58.768243 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acf95c20-2f78-4fe8-9ab7-974b33dd4ad8 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:58.783742 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:43:58.806090 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d9537d8-957d-4831-8644-ba245c6c7a66 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:58.806710 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d9537d8-957d-4831-8644-ba245c6c7a66 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:58.807047 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d9537d8-957d-4831-8644-ba245c6c7a66 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:58.807324 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d9537d8-957d-4831-8644-ba245c6c7a66 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:58.808093 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d9537d8-957d-4831-8644-ba245c6c7a66 None admin] RBAC: Adding query filter params (role.id=8e1192374afc4af6a7cea01c69e72402, scope.project.id=4562f3951fbe465280073e60a11e44e2, user.id=121c374d49a64335ab74a431cfbc1320) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:43:58.808415 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d9537d8-957d-4831-8644-ba245c6c7a66 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:58.809886 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d9537d8-957d-4831-8644-ba245c6c7a66 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:58.816080 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 450/897] 10.222.0.27 () {56 vars in 1374 bytes} [Tue Aug 30 01:43:58 2022] GET /identity/v3/role_assignments?user.id=121c374d49a64335ab74a431cfbc1320&scope.project.id=4562f3951fbe465280073e60a11e44e2&role.id=8e1192374afc4af6a7cea01c69e72402 => generated 259 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:43:58.883456 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bfc85d9-529b-4a65-87ba-ab96879be1af None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:58.921930 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d0c13f7-014d-4378-bfb1-4ca52b776f82 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:58.923040 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d0c13f7-014d-4378-bfb1-4ca52b776f82 None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:58.923414 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d0c13f7-014d-4378-bfb1-4ca52b776f82 None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:58.923808 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d0c13f7-014d-4378-bfb1-4ca52b776f82 None admin] PATH_INFO: `/v3/services/volumev3` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:58.924513 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d0c13f7-014d-4378-bfb1-4ca52b776f82 None admin] RBAC: Authorizing `identity:get_service(service_id=volumev3)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:58.926032 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d0c13f7-014d-4378-bfb1-4ca52b776f82 None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:58.936106 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application [None req-5d0c13f7-014d-4378-bfb1-4ca52b776f82 None admin] Could not find service: volumev3.: keystone.exception.ServiceNotFound: Could not find service: volumev3. Aug 30 01:43:58.936106 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 01:43:58.936106 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Aug 30 01:43:58.936106 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Aug 30 01:43:58.936106 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Aug 30 01:43:58.936106 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Aug 30 01:43:58.936106 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Aug 30 01:43:58.936106 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 30 01:43:58.936106 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Aug 30 01:43:58.936106 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application Aug 30 01:43:58.936106 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 30 01:43:58.936106 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application Aug 30 01:43:58.936106 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 01:43:58.936106 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 01:43:58.936106 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 01:43:58.936106 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 01:43:58.936106 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 01:43:58.937317 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 01:43:58.937317 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 01:43:58.937317 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 01:43:58.937317 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 01:43:58.937317 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 01:43:58.937317 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 01:43:58.937317 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Aug 30 01:43:58.937317 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._get_service(service_id) Aug 30 01:43:58.937317 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Aug 30 01:43:58.937317 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Aug 30 01:43:58.937317 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 01:43:58.937317 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 01:43:58.937317 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 01:43:58.937317 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 01:43:58.937317 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 01:43:58.937317 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 01:43:58.938201 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 01:43:58.938201 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application with Lock( Aug 30 01:43:58.938201 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 01:43:58.938201 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._enter() Aug 30 01:43:58.938201 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 01:43:58.938201 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 01:43:58.938201 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 01:43:58.938201 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.creator() Aug 30 01:43:58.938201 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 01:43:58.938201 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application created_value = creator( Aug 30 01:43:58.938201 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Aug 30 01:43:58.938201 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 30 01:43:58.938201 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Aug 30 01:43:58.938201 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application Aug 30 01:43:58.939393 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 448/898] 10.222.0.27 () {56 vars in 1114 bytes} [Tue Aug 30 01:43:58 2022] GET /identity/v3/services/volumev3 => generated 89 bytes in 172 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 01:43:58.946616 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d9537d8-957d-4831-8644-ba245c6c7a66 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:59.051808 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:43:59.061566 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d9537d8-957d-4831-8644-ba245c6c7a66 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:59.099981 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-149203ca-079b-4675-9be2-e1e757e58e71 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:59.100886 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-149203ca-079b-4675-9be2-e1e757e58e71 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:59.101626 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-149203ca-079b-4675-9be2-e1e757e58e71 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:59.102086 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-149203ca-079b-4675-9be2-e1e757e58e71 None admin] PATH_INFO: `/v3/services` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:59.102887 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-149203ca-079b-4675-9be2-e1e757e58e71 None admin] RBAC: Adding query filter params (name=volumev3) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:43:59.103375 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-149203ca-079b-4675-9be2-e1e757e58e71 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:59.104915 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-149203ca-079b-4675-9be2-e1e757e58e71 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:59.110382 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 451/899] 10.222.0.27 () {56 vars in 1114 bytes} [Tue Aug 30 01:43:58 2022] GET /identity/v3/services?name=volumev3 => generated 125 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:43:59.118773 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d0c13f7-014d-4378-bfb1-4ca52b776f82 None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:59.211983 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:43:59.237491 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d0c13f7-014d-4378-bfb1-4ca52b776f82 None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:59.242057 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5a0f3d0-b192-488b-abc1-9be05d014246 None None] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:59.242340 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5a0f3d0-b192-488b-abc1-9be05d014246 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:59.242616 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5a0f3d0-b192-488b-abc1-9be05d014246 None None] PATH_INFO: `/` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:59.244197 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 452/900] 10.222.0.27 () {54 vars in 927 bytes} [Tue Aug 30 01:43:59 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 01:43:59.249223 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ba06d4b-3fea-4989-a6f4-968df6e1d372 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:59.249525 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ba06d4b-3fea-4989-a6f4-968df6e1d372 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:59.249801 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ba06d4b-3fea-4989-a6f4-968df6e1d372 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:59.269374 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-6ba06d4b-3fea-4989-a6f4-968df6e1d372 None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 01:43:59.271334 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-6ba06d4b-3fea-4989-a6f4-968df6e1d372 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:43:59.274836 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72ff6999-fb6c-4f2e-9730-9937848d726a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:59.275493 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72ff6999-fb6c-4f2e-9730-9937848d726a None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:59.275785 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72ff6999-fb6c-4f2e-9730-9937848d726a None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:59.276057 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72ff6999-fb6c-4f2e-9730-9937848d726a None admin] PATH_INFO: `/v3/services` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:59.276694 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72ff6999-fb6c-4f2e-9730-9937848d726a None admin] RBAC: Adding query filter params (type=volumev3) {{(pid=81578) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:43:59.276997 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72ff6999-fb6c-4f2e-9730-9937848d726a None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:59.278391 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72ff6999-fb6c-4f2e-9730-9937848d726a None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:59.283963 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 449/901] 10.222.0.27 () {56 vars in 1114 bytes} [Tue Aug 30 01:43:59 2022] GET /identity/v3/services?type=volumev3 => generated 362 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:43:59.290267 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72ff6999-fb6c-4f2e-9730-9937848d726a None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:59.373703 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 453/902] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Aug 30 01:43:59 2022] POST /identity/v3/auth/tokens => generated 1372 bytes in 120 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Aug 30 01:43:59.412158 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72ff6999-fb6c-4f2e-9730-9937848d726a None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:59.448260 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3051114-e3b6-4a37-8f5a-97f07cdce9f8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:59.448872 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3051114-e3b6-4a37-8f5a-97f07cdce9f8 None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:59.449149 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3051114-e3b6-4a37-8f5a-97f07cdce9f8 None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:59.449417 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3051114-e3b6-4a37-8f5a-97f07cdce9f8 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:59.450148 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3051114-e3b6-4a37-8f5a-97f07cdce9f8 None admin] RBAC: Adding query filter params (interface=public, service_id=6515bb54cb1f40789d66fa6f48d46546, region_id=RegionOne) {{(pid=81578) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:43:59.450454 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3051114-e3b6-4a37-8f5a-97f07cdce9f8 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:59.451869 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3051114-e3b6-4a37-8f5a-97f07cdce9f8 None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:59.455173 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-142f7d5d-4556-4fa8-91f0-e6522390babf None None] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:59.455460 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-142f7d5d-4556-4fa8-91f0-e6522390babf None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:59.455735 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-142f7d5d-4556-4fa8-91f0-e6522390babf None None] PATH_INFO: `/` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:59.457673 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 450/904] 10.222.0.27 () {56 vars in 1251 bytes} [Tue Aug 30 01:43:59 2022] GET /identity/v3/endpoints?service_id=6515bb54cb1f40789d66fa6f48d46546&interface=public®ion_id=RegionOne => generated 194 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 30 01:43:59.458934 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 454/904] 10.222.0.27 () {54 vars in 927 bytes} [Tue Aug 30 01:43:59 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 30 01:43:59.467110 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74b6302d-bcc9-45fd-adaf-7602df6c3beb None None] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:59.467338 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74b6302d-bcc9-45fd-adaf-7602df6c3beb None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:59.467610 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74b6302d-bcc9-45fd-adaf-7602df6c3beb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:59.470355 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-142f7d5d-4556-4fa8-91f0-e6522390babf None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:59.474220 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:43:59.489733 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.auth.core [None req-74b6302d-bcc9-45fd-adaf-7602df6c3beb None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81578) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 01:43:59.491213 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-74b6302d-bcc9-45fd-adaf-7602df6c3beb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:43:59.569133 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 451/905] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Aug 30 01:43:59 2022] POST /identity/v3/auth/tokens => generated 1372 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Aug 30 01:43:59.580427 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:43:59.601024 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-142f7d5d-4556-4fa8-91f0-e6522390babf None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:59.638070 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c6d9eea-4ccb-4f92-946b-e38e61f26279 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:59.638684 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c6d9eea-4ccb-4f92-946b-e38e61f26279 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:59.639008 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c6d9eea-4ccb-4f92-946b-e38e61f26279 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:59.639340 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c6d9eea-4ccb-4f92-946b-e38e61f26279 None admin] PATH_INFO: `/v3/services` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:59.639949 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c6d9eea-4ccb-4f92-946b-e38e61f26279 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:59.641444 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c6d9eea-4ccb-4f92-946b-e38e61f26279 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:59.642499 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74b6302d-bcc9-45fd-adaf-7602df6c3beb None None] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:59.649806 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 455/906] 10.222.0.27 () {56 vars in 1087 bytes} [Tue Aug 30 01:43:59 2022] GET /identity/v3/services => generated 1268 bytes in 180 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 01:43:59.776827 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74b6302d-bcc9-45fd-adaf-7602df6c3beb None None] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:59.811150 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:43:59.816470 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce7176ac-476c-47c9-a875-28ee71731b84 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:59.817679 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce7176ac-476c-47c9-a875-28ee71731b84 None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:43:59.817964 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce7176ac-476c-47c9-a875-28ee71731b84 None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:43:59.818235 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce7176ac-476c-47c9-a875-28ee71731b84 None admin] PATH_INFO: `/v3/services/compute_legacy` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:43:59.818896 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce7176ac-476c-47c9-a875-28ee71731b84 None admin] RBAC: Authorizing `identity:get_service(service_id=compute_legacy)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:43:59.820188 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce7176ac-476c-47c9-a875-28ee71731b84 None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:43:59.829239 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application [None req-ce7176ac-476c-47c9-a875-28ee71731b84 None admin] Could not find service: compute_legacy.: keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Aug 30 01:43:59.829239 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 01:43:59.829239 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Aug 30 01:43:59.829239 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Aug 30 01:43:59.829239 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Aug 30 01:43:59.829239 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Aug 30 01:43:59.829239 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Aug 30 01:43:59.829239 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 30 01:43:59.829239 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Aug 30 01:43:59.829239 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application Aug 30 01:43:59.829239 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 30 01:43:59.829239 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application Aug 30 01:43:59.829239 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 01:43:59.829239 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 01:43:59.829239 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 01:43:59.829239 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 01:43:59.830230 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 01:43:59.830230 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 01:43:59.830230 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 01:43:59.830230 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 01:43:59.830230 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 01:43:59.830230 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 01:43:59.830230 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 01:43:59.830230 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Aug 30 01:43:59.830230 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._get_service(service_id) Aug 30 01:43:59.830230 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Aug 30 01:43:59.830230 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Aug 30 01:43:59.830230 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 01:43:59.830230 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 01:43:59.830230 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 01:43:59.830230 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 01:43:59.830230 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 01:43:59.830230 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 01:43:59.831166 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 01:43:59.831166 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application with Lock( Aug 30 01:43:59.831166 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 01:43:59.831166 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._enter() Aug 30 01:43:59.831166 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 01:43:59.831166 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 01:43:59.831166 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 01:43:59.831166 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.creator() Aug 30 01:43:59.831166 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 01:43:59.831166 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application created_value = creator( Aug 30 01:43:59.831166 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Aug 30 01:43:59.831166 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 30 01:43:59.831166 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Aug 30 01:43:59.831166 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application Aug 30 01:43:59.831166 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 452/907] 10.222.0.27 () {56 vars in 1132 bytes} [Tue Aug 30 01:43:59 2022] GET /identity/v3/services/compute_legacy => generated 95 bytes in 189 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 30 01:43:59.838366 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c6d9eea-4ccb-4f92-946b-e38e61f26279 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:59.848072 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce7176ac-476c-47c9-a875-28ee71731b84 None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:43:59.961620 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c6d9eea-4ccb-4f92-946b-e38e61f26279 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:59.967727 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce7176ac-476c-47c9-a875-28ee71731b84 None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:43:59.998698 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d7d332a-99dd-4cd0-94c5-887355ef35c0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:43:59.999526 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d7d332a-99dd-4cd0-94c5-887355ef35c0 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:44:00.000052 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d7d332a-99dd-4cd0-94c5-887355ef35c0 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:44:00.000553 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d7d332a-99dd-4cd0-94c5-887355ef35c0 None admin] PATH_INFO: `/v3/services` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:44:00.001508 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d7d332a-99dd-4cd0-94c5-887355ef35c0 None admin] RBAC: Adding query filter params (name=compute_legacy) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:44:00.002115 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d7d332a-99dd-4cd0-94c5-887355ef35c0 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:44:00.003197 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d3feb2d-4685-4734-b114-76dd98f1cca9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:44:00.003802 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d3feb2d-4685-4734-b114-76dd98f1cca9 None admin] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:44:00.004083 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d3feb2d-4685-4734-b114-76dd98f1cca9 None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:44:00.004242 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d7d332a-99dd-4cd0-94c5-887355ef35c0 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:44:00.004344 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d3feb2d-4685-4734-b114-76dd98f1cca9 None admin] PATH_INFO: `/v3/services` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:44:00.005573 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d3feb2d-4685-4734-b114-76dd98f1cca9 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:44:00.007050 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d3feb2d-4685-4734-b114-76dd98f1cca9 None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:44:00.015501 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 456/908] 10.222.0.27 () {56 vars in 1126 bytes} [Tue Aug 30 01:43:59 2022] GET /identity/v3/services?name=compute_legacy => generated 131 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:44:00.016769 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d7d332a-99dd-4cd0-94c5-887355ef35c0 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:44:00.038282 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:44:00.087841 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 453/909] 10.222.0.27 () {60 vars in 1141 bytes} [Tue Aug 30 01:43:59 2022] POST /identity/v3/services => generated 241 bytes in 240 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 30 01:44:00.140426 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d7d332a-99dd-4cd0-94c5-887355ef35c0 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:44:00.190581 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed2a3b7f-04de-4fd9-9a19-18704ca84520 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:44:00.191559 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed2a3b7f-04de-4fd9-9a19-18704ca84520 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:44:00.191998 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed2a3b7f-04de-4fd9-9a19-18704ca84520 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:44:00.192994 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed2a3b7f-04de-4fd9-9a19-18704ca84520 None admin] PATH_INFO: `/v3/services` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:44:00.193892 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed2a3b7f-04de-4fd9-9a19-18704ca84520 None admin] RBAC: Adding query filter params (type=compute_legacy) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:44:00.194344 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed2a3b7f-04de-4fd9-9a19-18704ca84520 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:44:00.195945 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed2a3b7f-04de-4fd9-9a19-18704ca84520 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:44:00.202237 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 457/910] 10.222.0.27 () {56 vars in 1126 bytes} [Tue Aug 30 01:44:00 2022] GET /identity/v3/services?type=compute_legacy => generated 386 bytes in 186 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 30 01:44:00.208880 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d3feb2d-4685-4734-b114-76dd98f1cca9 None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:44:00.273089 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:44:00.320068 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d3feb2d-4685-4734-b114-76dd98f1cca9 None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:44:00.355612 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9943952-1f50-4380-8b60-74885e116738 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:44:00.356221 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9943952-1f50-4380-8b60-74885e116738 None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:44:00.356500 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9943952-1f50-4380-8b60-74885e116738 None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:44:00.356771 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9943952-1f50-4380-8b60-74885e116738 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:44:00.357484 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9943952-1f50-4380-8b60-74885e116738 None admin] RBAC: Adding query filter params (interface=public, service_id=cd440efa579c46b2a2c2de0debfc4bf1, region_id=RegionOne) {{(pid=81578) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:44:00.357806 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9943952-1f50-4380-8b60-74885e116738 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:44:00.359160 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9943952-1f50-4380-8b60-74885e116738 None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:44:00.365168 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 454/911] 10.222.0.27 () {56 vars in 1251 bytes} [Tue Aug 30 01:44:00 2022] GET /identity/v3/endpoints?service_id=cd440efa579c46b2a2c2de0debfc4bf1&interface=public®ion_id=RegionOne => generated 194 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:44:00.370692 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed2a3b7f-04de-4fd9-9a19-18704ca84520 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:44:00.452918 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:44:00.477314 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed2a3b7f-04de-4fd9-9a19-18704ca84520 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:44:00.514716 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61e72946-e2ab-4b40-a2e0-6839489ab394 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:44:00.515326 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61e72946-e2ab-4b40-a2e0-6839489ab394 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:44:00.515605 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61e72946-e2ab-4b40-a2e0-6839489ab394 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:44:00.515877 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61e72946-e2ab-4b40-a2e0-6839489ab394 None admin] PATH_INFO: `/v3/services` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:44:00.516483 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61e72946-e2ab-4b40-a2e0-6839489ab394 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:44:00.517876 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61e72946-e2ab-4b40-a2e0-6839489ab394 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:44:00.524311 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 458/912] 10.222.0.27 () {56 vars in 1087 bytes} [Tue Aug 30 01:44:00 2022] GET /identity/v3/services => generated 1497 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 01:44:00.905467 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc663ce5-36c0-4b6f-b40b-5213aac24126 None None] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:44:00.905996 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc663ce5-36c0-4b6f-b40b-5213aac24126 None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:44:00.905996 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc663ce5-36c0-4b6f-b40b-5213aac24126 None None] PATH_INFO: `/` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:44:00.907614 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 455/913] 10.222.0.27 () {54 vars in 927 bytes} [Tue Aug 30 01:44:00 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 01:44:00.913489 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfb591b5-af23-411d-a5d8-2bdd01327b54 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:44:00.913784 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfb591b5-af23-411d-a5d8-2bdd01327b54 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:44:00.914065 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfb591b5-af23-411d-a5d8-2bdd01327b54 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:44:00.934734 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-dfb591b5-af23-411d-a5d8-2bdd01327b54 None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 01:44:00.935927 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-dfb591b5-af23-411d-a5d8-2bdd01327b54 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:44:01.006435 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 459/914] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Aug 30 01:44:00 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 30 01:44:01.295157 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc663ce5-36c0-4b6f-b40b-5213aac24126 None None] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:44:01.405747 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc663ce5-36c0-4b6f-b40b-5213aac24126 None None] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:44:01.428522 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:44:01.443198 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aace42d9-f258-408d-99b6-4b01e0cf6188 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:44:01.444037 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aace42d9-f258-408d-99b6-4b01e0cf6188 None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:44:01.444469 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aace42d9-f258-408d-99b6-4b01e0cf6188 None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:44:01.444886 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aace42d9-f258-408d-99b6-4b01e0cf6188 None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:44:01.455855 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aace42d9-f258-408d-99b6-4b01e0cf6188 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 30 01:44:01.462181 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application [None req-aace42d9-f258-408d-99b6-4b01e0cf6188 None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Aug 30 01:44:01.462181 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 01:44:01.462181 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 01:44:01.462181 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 01:44:01.462181 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 01:44:01.462181 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 01:44:01.462181 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 01:44:01.462181 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 01:44:01.462181 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 01:44:01.462181 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 01:44:01.462181 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 01:44:01.462181 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 01:44:01.462181 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 30 01:44:01.462181 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 30 01:44:01.462181 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 30 01:44:01.462181 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 30 01:44:01.463347 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 30 01:44:01.463347 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 30 01:44:01.463347 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 30 01:44:01.463347 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 30 01:44:01.463347 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 01:44:01.463347 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 01:44:01.463347 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 01:44:01.463347 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 01:44:01.463347 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 01:44:01.463347 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 01:44:01.463347 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 01:44:01.463347 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application with Lock( Aug 30 01:44:01.463347 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 01:44:01.463347 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._enter() Aug 30 01:44:01.463347 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 01:44:01.463347 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 01:44:01.464418 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 01:44:01.464418 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.creator() Aug 30 01:44:01.464418 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 01:44:01.464418 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application created_value = creator( Aug 30 01:44:01.464418 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 30 01:44:01.464418 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 30 01:44:01.464418 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 30 01:44:01.464418 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 30 01:44:01.464418 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 30 01:44:01.464418 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 30 01:44:01.464418 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Aug 30 01:44:01.464418 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application Aug 30 01:44:01.466675 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 456/915] 10.222.0.27 () {56 vars in 1120 bytes} [Tue Aug 30 01:44:01 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 172 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 01:44:01.472642 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfb591b5-af23-411d-a5d8-2bdd01327b54 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:44:01.549068 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:44:01.581066 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfb591b5-af23-411d-a5d8-2bdd01327b54 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:44:01.618954 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3cfce20-00d5-482a-9155-a363e6e54411 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:44:01.619566 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3cfce20-00d5-482a-9155-a363e6e54411 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:44:01.619848 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3cfce20-00d5-482a-9155-a363e6e54411 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:44:01.620122 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3cfce20-00d5-482a-9155-a363e6e54411 None admin] PATH_INFO: `/v3/roles` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:44:01.620795 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3cfce20-00d5-482a-9155-a363e6e54411 None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:44:01.621105 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3cfce20-00d5-482a-9155-a363e6e54411 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:44:01.622505 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3cfce20-00d5-482a-9155-a363e6e54411 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:44:01.631813 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 460/916] 10.222.0.27 () {56 vars in 1115 bytes} [Tue Aug 30 01:44:01 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:44:01.637313 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aace42d9-f258-408d-99b6-4b01e0cf6188 None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:44:01.749987 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aace42d9-f258-408d-99b6-4b01e0cf6188 None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:44:01.762270 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fced6aa5-a72c-4a35-8c8c-10228dc322f0 None None] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:44:01.762557 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fced6aa5-a72c-4a35-8c8c-10228dc322f0 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:44:01.762831 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fced6aa5-a72c-4a35-8c8c-10228dc322f0 None None] PATH_INFO: `/` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:44:01.764725 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 461/917] 10.222.0.27 () {54 vars in 927 bytes} [Tue Aug 30 01:44:01 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 01:44:01.770290 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dea363a2-4fc6-44da-94d3-a1196a312533 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:44:01.770922 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dea363a2-4fc6-44da-94d3-a1196a312533 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:44:01.771445 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dea363a2-4fc6-44da-94d3-a1196a312533 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:44:01.788494 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c04ab050-c9ae-4897-88c0-2c12f0687e1c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:44:01.789376 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c04ab050-c9ae-4897-88c0-2c12f0687e1c None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:44:01.789822 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c04ab050-c9ae-4897-88c0-2c12f0687e1c None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:44:01.790236 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c04ab050-c9ae-4897-88c0-2c12f0687e1c None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:44:01.791441 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-dea363a2-4fc6-44da-94d3-a1196a312533 None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 01:44:01.792893 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-dea363a2-4fc6-44da-94d3-a1196a312533 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:44:01.804468 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c04ab050-c9ae-4897-88c0-2c12f0687e1c None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:44:01.805874 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c04ab050-c9ae-4897-88c0-2c12f0687e1c None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:44:01.821674 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application [None req-c04ab050-c9ae-4897-88c0-2c12f0687e1c None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 01:44:01.821674 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 01:44:01.821674 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 01:44:01.821674 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 01:44:01.821674 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 01:44:01.821674 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 01:44:01.821674 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 01:44:01.821674 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 01:44:01.821674 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 01:44:01.821674 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 01:44:01.821674 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 01:44:01.821674 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 01:44:01.821674 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 30 01:44:01.821674 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 30 01:44:01.821674 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 30 01:44:01.821674 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 30 01:44:01.822656 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 01:44:01.822656 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 01:44:01.822656 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 01:44:01.822656 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 01:44:01.822656 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 01:44:01.822656 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 01:44:01.822656 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 01:44:01.822656 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application with Lock( Aug 30 01:44:01.822656 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 01:44:01.822656 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._enter() Aug 30 01:44:01.822656 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 01:44:01.822656 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 01:44:01.822656 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 01:44:01.822656 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.creator() Aug 30 01:44:01.822656 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 01:44:01.822656 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application created_value = creator( Aug 30 01:44:01.822656 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 30 01:44:01.823575 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 30 01:44:01.823575 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 01:44:01.823575 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application Aug 30 01:44:01.827509 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 457/918] 10.222.0.27 () {56 vars in 1108 bytes} [Tue Aug 30 01:44:01 2022] GET /identity/v3/domains/Default => generated 87 bytes in 190 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 01:44:01.828357 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c04ab050-c9ae-4897-88c0-2c12f0687e1c None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:44:01.861980 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:44:01.891473 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 462/919] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Aug 30 01:44:01 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 122 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 30 01:44:01.944956 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c04ab050-c9ae-4897-88c0-2c12f0687e1c None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:44:01.980476 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b053d574-d9fd-40ba-ab88-08751a73ee98 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:44:01.981083 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b053d574-d9fd-40ba-ab88-08751a73ee98 None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:44:01.981357 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b053d574-d9fd-40ba-ab88-08751a73ee98 None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:44:01.981653 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b053d574-d9fd-40ba-ab88-08751a73ee98 None admin] PATH_INFO: `/v3/domains` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:44:01.982375 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b053d574-d9fd-40ba-ab88-08751a73ee98 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81578) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:44:01.982677 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b053d574-d9fd-40ba-ab88-08751a73ee98 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:44:01.984124 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b053d574-d9fd-40ba-ab88-08751a73ee98 None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:44:01.997144 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 458/920] 10.222.0.27 () {56 vars in 1109 bytes} [Tue Aug 30 01:44:01 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:44:02.002682 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dea363a2-4fc6-44da-94d3-a1196a312533 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:44:02.071170 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:44:02.116081 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dea363a2-4fc6-44da-94d3-a1196a312533 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:44:02.138483 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0074fef9-35cb-44b3-a6eb-22bf1dc06f7e None None] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:44:02.138812 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0074fef9-35cb-44b3-a6eb-22bf1dc06f7e None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:44:02.139018 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0074fef9-35cb-44b3-a6eb-22bf1dc06f7e None None] PATH_INFO: `/` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:44:02.140633 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 459/921] 10.222.0.27 () {54 vars in 927 bytes} [Tue Aug 30 01:44:02 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 30 01:44:02.147314 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36842fbb-2123-4b1b-8ce5-67ad3cf8954d None None] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:44:02.147732 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36842fbb-2123-4b1b-8ce5-67ad3cf8954d None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:44:02.148153 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36842fbb-2123-4b1b-8ce5-67ad3cf8954d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:44:02.152803 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bb10b9e-eba4-4cbd-ac24-036e0f3260eb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:44:02.153437 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bb10b9e-eba4-4cbd-ac24-036e0f3260eb None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:44:02.153755 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bb10b9e-eba4-4cbd-ac24-036e0f3260eb None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:44:02.154030 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bb10b9e-eba4-4cbd-ac24-036e0f3260eb None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:44:02.161558 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3bb10b9e-eba4-4cbd-ac24-036e0f3260eb None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:44:02.163154 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3bb10b9e-eba4-4cbd-ac24-036e0f3260eb None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:44:02.167896 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.auth.core [None req-36842fbb-2123-4b1b-8ce5-67ad3cf8954d None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81578) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 01:44:02.169082 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-36842fbb-2123-4b1b-8ce5-67ad3cf8954d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:44:02.175263 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application [None req-3bb10b9e-eba4-4cbd-ac24-036e0f3260eb None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Aug 30 01:44:02.175263 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 01:44:02.175263 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 01:44:02.175263 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 01:44:02.175263 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 01:44:02.175263 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 01:44:02.175263 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 01:44:02.175263 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 01:44:02.175263 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 01:44:02.175263 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 01:44:02.175263 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 01:44:02.175263 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 01:44:02.175263 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 30 01:44:02.175263 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 30 01:44:02.175263 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 30 01:44:02.175263 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 30 01:44:02.176221 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 01:44:02.176221 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 01:44:02.176221 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 30 01:44:02.176221 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 30 01:44:02.176221 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 30 01:44:02.176221 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 30 01:44:02.176221 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 01:44:02.176221 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 01:44:02.176221 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 01:44:02.176221 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 01:44:02.176221 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 01:44:02.176221 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application with Lock( Aug 30 01:44:02.176221 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 01:44:02.176221 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._enter() Aug 30 01:44:02.176221 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 01:44:02.176221 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 01:44:02.176221 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 01:44:02.176221 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.creator() Aug 30 01:44:02.177303 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 01:44:02.177303 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application created_value = creator( Aug 30 01:44:02.177303 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 30 01:44:02.177303 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 30 01:44:02.177303 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 30 01:44:02.177303 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 30 01:44:02.177303 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 30 01:44:02.177303 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 30 01:44:02.177303 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Aug 30 01:44:02.177303 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Aug 30 01:44:02.177303 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 463/922] 10.222.0.27 () {56 vars in 1117 bytes} [Tue Aug 30 01:44:02 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 174 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 01:44:02.177970 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bb10b9e-eba4-4cbd-ac24-036e0f3260eb None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:44:02.248605 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 460/923] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Aug 30 01:44:02 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Aug 30 01:44:02.249840 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36842fbb-2123-4b1b-8ce5-67ad3cf8954d None None] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:44:02.277902 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:44:02.313938 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bb10b9e-eba4-4cbd-ac24-036e0f3260eb None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:44:02.345994 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:44:02.361192 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-426fdc48-0844-4031-a234-65aa7d97b5d8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:44:02.361873 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-426fdc48-0844-4031-a234-65aa7d97b5d8 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:44:02.362156 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-426fdc48-0844-4031-a234-65aa7d97b5d8 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:44:02.362431 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-426fdc48-0844-4031-a234-65aa7d97b5d8 None admin] PATH_INFO: `/v3/services/volumev3` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:44:02.363560 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-426fdc48-0844-4031-a234-65aa7d97b5d8 None admin] RBAC: Authorizing `identity:get_service(service_id=volumev3)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:44:02.365577 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-426fdc48-0844-4031-a234-65aa7d97b5d8 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:44:02.377787 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36842fbb-2123-4b1b-8ce5-67ad3cf8954d None None] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:44:02.378031 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application [None req-426fdc48-0844-4031-a234-65aa7d97b5d8 None admin] Could not find service: volumev3.: keystone.exception.ServiceNotFound: Could not find service: volumev3. Aug 30 01:44:02.378031 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 01:44:02.378031 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Aug 30 01:44:02.378031 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Aug 30 01:44:02.378031 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Aug 30 01:44:02.378031 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Aug 30 01:44:02.378031 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Aug 30 01:44:02.378031 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 30 01:44:02.378031 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Aug 30 01:44:02.378031 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Aug 30 01:44:02.378031 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 30 01:44:02.378031 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Aug 30 01:44:02.378031 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 01:44:02.378031 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 01:44:02.378031 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 01:44:02.378031 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 01:44:02.378031 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 01:44:02.379026 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 01:44:02.379026 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 01:44:02.379026 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 01:44:02.379026 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 01:44:02.379026 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 01:44:02.379026 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 01:44:02.379026 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Aug 30 01:44:02.379026 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_service(service_id) Aug 30 01:44:02.379026 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Aug 30 01:44:02.379026 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Aug 30 01:44:02.379026 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 01:44:02.379026 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 01:44:02.379026 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 01:44:02.379026 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 01:44:02.379026 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 01:44:02.379026 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 01:44:02.385562 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 01:44:02.385562 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application with Lock( Aug 30 01:44:02.385562 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 01:44:02.385562 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._enter() Aug 30 01:44:02.385562 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 01:44:02.385562 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 01:44:02.385562 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 01:44:02.385562 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.creator() Aug 30 01:44:02.385562 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 01:44:02.385562 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application created_value = creator( Aug 30 01:44:02.385562 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Aug 30 01:44:02.385562 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 30 01:44:02.385562 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Aug 30 01:44:02.385562 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Aug 30 01:44:02.385562 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 464/924] 10.222.0.27 () {56 vars in 1114 bytes} [Tue Aug 30 01:44:02 2022] GET /identity/v3/services/volumev3 => generated 89 bytes in 202 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 01:44:02.388082 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-426fdc48-0844-4031-a234-65aa7d97b5d8 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:44:02.422900 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4f40c7c-e85c-4491-9bf8-4dde34ea0952 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:44:02.423540 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4f40c7c-e85c-4491-9bf8-4dde34ea0952 None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:44:02.423823 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4f40c7c-e85c-4491-9bf8-4dde34ea0952 None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:44:02.424094 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4f40c7c-e85c-4491-9bf8-4dde34ea0952 None admin] PATH_INFO: `/v3/users` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:44:02.424833 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4f40c7c-e85c-4491-9bf8-4dde34ea0952 None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=81578) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:44:02.425370 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4f40c7c-e85c-4491-9bf8-4dde34ea0952 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:44:02.426674 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4f40c7c-e85c-4491-9bf8-4dde34ea0952 None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:44:02.445533 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 461/925] 10.222.0.27 () {56 vars in 1149 bytes} [Tue Aug 30 01:44:02 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 196 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:44:02.450663 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4f40c7c-e85c-4491-9bf8-4dde34ea0952 None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:44:02.498732 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-426fdc48-0844-4031-a234-65aa7d97b5d8 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:44:02.535460 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a99b26d-a40b-46fb-8f56-f8fdfd0c9282 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:44:02.536270 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a99b26d-a40b-46fb-8f56-f8fdfd0c9282 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:44:02.536552 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a99b26d-a40b-46fb-8f56-f8fdfd0c9282 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:44:02.536823 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a99b26d-a40b-46fb-8f56-f8fdfd0c9282 None admin] PATH_INFO: `/v3/services` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:44:02.537481 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a99b26d-a40b-46fb-8f56-f8fdfd0c9282 None admin] RBAC: Adding query filter params (name=volumev3) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:44:02.537791 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a99b26d-a40b-46fb-8f56-f8fdfd0c9282 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:44:02.539158 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a99b26d-a40b-46fb-8f56-f8fdfd0c9282 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:44:02.544055 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 465/926] 10.222.0.27 () {56 vars in 1114 bytes} [Tue Aug 30 01:44:02 2022] GET /identity/v3/services?name=volumev3 => generated 125 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 30 01:44:02.544837 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a99b26d-a40b-46fb-8f56-f8fdfd0c9282 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:44:02.546023 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:44:02.574172 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4f40c7c-e85c-4491-9bf8-4dde34ea0952 None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:44:02.597258 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:44:02.611354 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-577e6280-f7a0-4152-bd22-d3b3409d001b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:44:02.611982 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-577e6280-f7a0-4152-bd22-d3b3409d001b None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:44:02.612258 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-577e6280-f7a0-4152-bd22-d3b3409d001b None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:44:02.612522 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-577e6280-f7a0-4152-bd22-d3b3409d001b None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:44:02.623091 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-577e6280-f7a0-4152-bd22-d3b3409d001b None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:44:02.624531 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-577e6280-f7a0-4152-bd22-d3b3409d001b None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:44:02.639714 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application [None req-577e6280-f7a0-4152-bd22-d3b3409d001b None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 01:44:02.639714 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 01:44:02.639714 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 01:44:02.639714 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 01:44:02.639714 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 01:44:02.639714 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 01:44:02.639714 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 01:44:02.639714 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 01:44:02.639714 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 01:44:02.639714 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 01:44:02.639714 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 01:44:02.639714 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 01:44:02.639714 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 30 01:44:02.639714 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 30 01:44:02.639714 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 30 01:44:02.639714 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 30 01:44:02.640859 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 01:44:02.640859 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 01:44:02.640859 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 01:44:02.640859 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 01:44:02.640859 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 01:44:02.640859 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 01:44:02.640859 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 01:44:02.640859 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application with Lock( Aug 30 01:44:02.640859 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 01:44:02.640859 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._enter() Aug 30 01:44:02.640859 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 01:44:02.640859 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 01:44:02.640859 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 01:44:02.640859 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.creator() Aug 30 01:44:02.640859 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 01:44:02.640859 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application created_value = creator( Aug 30 01:44:02.640859 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 30 01:44:02.641904 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 30 01:44:02.641904 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 01:44:02.641904 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application Aug 30 01:44:02.641904 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 462/927] 10.222.0.27 () {56 vars in 1108 bytes} [Tue Aug 30 01:44:02 2022] GET /identity/v3/domains/Default => generated 87 bytes in 191 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 30 01:44:02.642151 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-577e6280-f7a0-4152-bd22-d3b3409d001b None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:44:02.650109 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a99b26d-a40b-46fb-8f56-f8fdfd0c9282 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:44:02.684394 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23716dd0-46ff-47ff-bafb-16b7551afe1f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:44:02.685023 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23716dd0-46ff-47ff-bafb-16b7551afe1f None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:44:02.685302 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23716dd0-46ff-47ff-bafb-16b7551afe1f None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:44:02.685678 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23716dd0-46ff-47ff-bafb-16b7551afe1f None admin] PATH_INFO: `/v3/services/object-store` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:44:02.686196 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23716dd0-46ff-47ff-bafb-16b7551afe1f None admin] RBAC: Authorizing `identity:get_service(service_id=object-store)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:44:02.687559 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23716dd0-46ff-47ff-bafb-16b7551afe1f None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:44:02.697036 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application [None req-23716dd0-46ff-47ff-bafb-16b7551afe1f None admin] Could not find service: object-store.: keystone.exception.ServiceNotFound: Could not find service: object-store. Aug 30 01:44:02.697036 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 01:44:02.697036 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Aug 30 01:44:02.697036 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Aug 30 01:44:02.697036 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Aug 30 01:44:02.697036 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Aug 30 01:44:02.697036 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Aug 30 01:44:02.697036 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 30 01:44:02.697036 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Aug 30 01:44:02.697036 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Aug 30 01:44:02.697036 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 30 01:44:02.697036 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Aug 30 01:44:02.697036 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 01:44:02.697036 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 01:44:02.697036 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 01:44:02.697036 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 01:44:02.697036 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 01:44:02.698247 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 01:44:02.698247 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 01:44:02.698247 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 01:44:02.698247 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 01:44:02.698247 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 01:44:02.698247 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 01:44:02.698247 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Aug 30 01:44:02.698247 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_service(service_id) Aug 30 01:44:02.698247 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Aug 30 01:44:02.698247 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Aug 30 01:44:02.698247 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 01:44:02.698247 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 01:44:02.698247 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 01:44:02.698247 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 01:44:02.698247 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 01:44:02.698247 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 01:44:02.699107 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 01:44:02.699107 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application with Lock( Aug 30 01:44:02.699107 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 01:44:02.699107 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._enter() Aug 30 01:44:02.699107 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 01:44:02.699107 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 01:44:02.699107 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 01:44:02.699107 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.creator() Aug 30 01:44:02.699107 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 01:44:02.699107 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application created_value = creator( Aug 30 01:44:02.699107 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Aug 30 01:44:02.699107 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 30 01:44:02.699107 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Aug 30 01:44:02.699107 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Aug 30 01:44:02.699107 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 466/928] 10.222.0.27 () {56 vars in 1126 bytes} [Tue Aug 30 01:44:02 2022] GET /identity/v3/services/object-store => generated 93 bytes in 154 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 01:44:02.699922 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4642509b-b03e-47d2-a192-26d76c867229 None None] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:44:02.699922 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4642509b-b03e-47d2-a192-26d76c867229 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:44:02.700177 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4642509b-b03e-47d2-a192-26d76c867229 None None] PATH_INFO: `/` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:44:02.705215 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 467/929] 10.222.0.27 () {54 vars in 927 bytes} [Tue Aug 30 01:44:02 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 01:44:02.705215 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4642509b-b03e-47d2-a192-26d76c867229 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:44:02.751153 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-577e6280-f7a0-4152-bd22-d3b3409d001b None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:44:02.780987 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:44:02.786645 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13298406-ba20-44d6-80df-3d43e53110e2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:44:02.787241 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13298406-ba20-44d6-80df-3d43e53110e2 None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:44:02.787518 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13298406-ba20-44d6-80df-3d43e53110e2 None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:44:02.787783 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13298406-ba20-44d6-80df-3d43e53110e2 None admin] PATH_INFO: `/v3/services` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:44:02.788407 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13298406-ba20-44d6-80df-3d43e53110e2 None admin] RBAC: Adding query filter params (type=volumev3) {{(pid=81578) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:44:02.788708 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13298406-ba20-44d6-80df-3d43e53110e2 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:44:02.790079 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13298406-ba20-44d6-80df-3d43e53110e2 None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:44:02.794768 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 463/930] 10.222.0.27 () {56 vars in 1114 bytes} [Tue Aug 30 01:44:02 2022] GET /identity/v3/services?type=volumev3 => generated 362 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:44:02.795534 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13298406-ba20-44d6-80df-3d43e53110e2 None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:44:02.813289 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4642509b-b03e-47d2-a192-26d76c867229 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:44:02.827394 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:44:02.847181 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6646315-9999-4924-bb00-b4b559f9f719 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:44:02.847826 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6646315-9999-4924-bb00-b4b559f9f719 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:44:02.848054 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6646315-9999-4924-bb00-b4b559f9f719 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:44:02.848425 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6646315-9999-4924-bb00-b4b559f9f719 None admin] PATH_INFO: `/v3/domains` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:44:02.849058 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6646315-9999-4924-bb00-b4b559f9f719 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:44:02.849613 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6646315-9999-4924-bb00-b4b559f9f719 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:44:02.850811 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6646315-9999-4924-bb00-b4b559f9f719 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:44:02.863234 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 468/931] 10.222.0.27 () {56 vars in 1109 bytes} [Tue Aug 30 01:44:02 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:44:02.864727 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-351c97a6-1612-4e8d-972c-a918c087f64f None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:44:02.865004 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-351c97a6-1612-4e8d-972c-a918c087f64f None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:44:02.865270 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-351c97a6-1612-4e8d-972c-a918c087f64f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:44:02.882760 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-351c97a6-1612-4e8d-972c-a918c087f64f None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 01:44:02.883941 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-351c97a6-1612-4e8d-972c-a918c087f64f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:44:02.902430 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13298406-ba20-44d6-80df-3d43e53110e2 None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:44:02.935058 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b90be774-9093-45e5-bc4f-1e9a41f41361 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:44:02.935655 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b90be774-9093-45e5-bc4f-1e9a41f41361 None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:44:02.935931 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b90be774-9093-45e5-bc4f-1e9a41f41361 None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:44:02.936197 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b90be774-9093-45e5-bc4f-1e9a41f41361 None admin] PATH_INFO: `/v3/services` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:44:02.936820 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b90be774-9093-45e5-bc4f-1e9a41f41361 None admin] RBAC: Adding query filter params (name=object-store) {{(pid=81578) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:44:02.937119 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b90be774-9093-45e5-bc4f-1e9a41f41361 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:44:02.938500 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b90be774-9093-45e5-bc4f-1e9a41f41361 None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:44:02.943648 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 464/932] 10.222.0.27 () {56 vars in 1122 bytes} [Tue Aug 30 01:44:02 2022] GET /identity/v3/services?name=object-store => generated 129 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:44:02.944393 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b90be774-9093-45e5-bc4f-1e9a41f41361 None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:44:02.954910 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 469/933] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Aug 30 01:44:02 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 30 01:44:02.955735 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-351c97a6-1612-4e8d-972c-a918c087f64f None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:44:03.023769 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:44:03.064360 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-351c97a6-1612-4e8d-972c-a918c087f64f None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:44:03.064956 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b90be774-9093-45e5-bc4f-1e9a41f41361 None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:44:03.068667 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:44:03.101246 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40f8e03f-a025-4e49-afb4-131bb4471a1c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:44:03.101742 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd6e5a86-5388-4c1d-be51-91aa7052d65b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:44:03.102168 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40f8e03f-a025-4e49-afb4-131bb4471a1c None admin] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:44:03.102246 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd6e5a86-5388-4c1d-be51-91aa7052d65b None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:44:03.102317 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40f8e03f-a025-4e49-afb4-131bb4471a1c None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:44:03.102398 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd6e5a86-5388-4c1d-be51-91aa7052d65b None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:44:03.102681 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40f8e03f-a025-4e49-afb4-131bb4471a1c None admin] PATH_INFO: `/v3/endpoints` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:44:03.102780 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd6e5a86-5388-4c1d-be51-91aa7052d65b None admin] PATH_INFO: `/v3/projects/service` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:44:03.103216 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40f8e03f-a025-4e49-afb4-131bb4471a1c None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:44:03.104592 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40f8e03f-a025-4e49-afb4-131bb4471a1c None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:44:03.109827 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd6e5a86-5388-4c1d-be51-91aa7052d65b None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:44:03.111270 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd6e5a86-5388-4c1d-be51-91aa7052d65b None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:44:03.122913 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application [None req-cd6e5a86-5388-4c1d-be51-91aa7052d65b None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 30 01:44:03.122913 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 01:44:03.122913 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 01:44:03.122913 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 01:44:03.122913 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 01:44:03.122913 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 01:44:03.122913 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 01:44:03.122913 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 01:44:03.122913 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 01:44:03.122913 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 01:44:03.122913 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 01:44:03.122913 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 01:44:03.122913 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 30 01:44:03.122913 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 30 01:44:03.122913 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 30 01:44:03.123999 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 30 01:44:03.123999 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 01:44:03.123999 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 01:44:03.123999 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 01:44:03.123999 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 01:44:03.123999 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 01:44:03.123999 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 01:44:03.123999 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 01:44:03.123999 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application with Lock( Aug 30 01:44:03.123999 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 01:44:03.123999 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._enter() Aug 30 01:44:03.123999 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 01:44:03.123999 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 01:44:03.123999 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 01:44:03.123999 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.creator() Aug 30 01:44:03.123999 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 01:44:03.123999 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application created_value = creator( Aug 30 01:44:03.124883 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 30 01:44:03.124883 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 30 01:44:03.124883 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 30 01:44:03.124883 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 30 01:44:03.124883 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 30 01:44:03.124883 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 30 01:44:03.124883 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 30 01:44:03.124883 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Aug 30 01:44:03.124883 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 470/934] 10.222.0.27 () {56 vars in 1111 bytes} [Tue Aug 30 01:44:02 2022] GET /identity/v3/projects/service => generated 88 bytes in 169 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 01:44:03.129933 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd6e5a86-5388-4c1d-be51-91aa7052d65b None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:44:03.180929 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 465/935] 10.222.0.27 () {60 vars in 1144 bytes} [Tue Aug 30 01:44:02 2022] POST /identity/v3/endpoints => generated 346 bytes in 237 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 01:44:03.182119 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40f8e03f-a025-4e49-afb4-131bb4471a1c None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:44:03.249762 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd6e5a86-5388-4c1d-be51-91aa7052d65b None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:44:03.296869 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0eec7c1c-5d44-41e2-bb8a-e0af1a6ebf23 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:44:03.297520 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0eec7c1c-5d44-41e2-bb8a-e0af1a6ebf23 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:44:03.297962 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0eec7c1c-5d44-41e2-bb8a-e0af1a6ebf23 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:44:03.298256 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0eec7c1c-5d44-41e2-bb8a-e0af1a6ebf23 None admin] PATH_INFO: `/v3/services` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:44:03.298893 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0eec7c1c-5d44-41e2-bb8a-e0af1a6ebf23 None admin] RBAC: Adding query filter params (type=object-store) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:44:03.299196 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0eec7c1c-5d44-41e2-bb8a-e0af1a6ebf23 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:44:03.300599 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0eec7c1c-5d44-41e2-bb8a-e0af1a6ebf23 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:44:03.311400 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 471/936] 10.222.0.27 () {56 vars in 1122 bytes} [Tue Aug 30 01:44:03 2022] GET /identity/v3/services?type=object-store => generated 356 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:44:03.311400 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0eec7c1c-5d44-41e2-bb8a-e0af1a6ebf23 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:44:03.316642 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40f8e03f-a025-4e49-afb4-131bb4471a1c None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:44:03.338040 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:44:03.339321 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:44:03.358951 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2476a19-9232-4800-ac4b-b5aabe2d1850 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:44:03.359489 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2476a19-9232-4800-ac4b-b5aabe2d1850 None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:44:03.359769 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2476a19-9232-4800-ac4b-b5aabe2d1850 None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:44:03.360041 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2476a19-9232-4800-ac4b-b5aabe2d1850 None admin] PATH_INFO: `/v3/projects` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:44:03.360736 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2476a19-9232-4800-ac4b-b5aabe2d1850 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81578) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:44:03.361037 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2476a19-9232-4800-ac4b-b5aabe2d1850 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:44:03.362490 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2476a19-9232-4800-ac4b-b5aabe2d1850 None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:44:03.377201 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 466/937] 10.222.0.27 () {56 vars in 1148 bytes} [Tue Aug 30 01:44:03 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 195 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:44:03.378087 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2476a19-9232-4800-ac4b-b5aabe2d1850 None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:44:03.426886 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0eec7c1c-5d44-41e2-bb8a-e0af1a6ebf23 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:44:03.461719 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a21bc4d-7681-4292-9f8b-28d3fa4d7960 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:44:03.462349 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a21bc4d-7681-4292-9f8b-28d3fa4d7960 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:44:03.462633 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a21bc4d-7681-4292-9f8b-28d3fa4d7960 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:44:03.462907 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a21bc4d-7681-4292-9f8b-28d3fa4d7960 None admin] PATH_INFO: `/v3/services/compute_legacy` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:44:03.463495 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a21bc4d-7681-4292-9f8b-28d3fa4d7960 None admin] RBAC: Authorizing `identity:get_service(service_id=compute_legacy)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:44:03.464856 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a21bc4d-7681-4292-9f8b-28d3fa4d7960 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:44:03.473887 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application [None req-4a21bc4d-7681-4292-9f8b-28d3fa4d7960 None admin] Could not find service: compute_legacy.: keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Aug 30 01:44:03.473887 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 01:44:03.473887 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Aug 30 01:44:03.473887 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Aug 30 01:44:03.473887 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Aug 30 01:44:03.473887 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Aug 30 01:44:03.473887 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Aug 30 01:44:03.473887 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 30 01:44:03.473887 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Aug 30 01:44:03.473887 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Aug 30 01:44:03.473887 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 30 01:44:03.473887 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Aug 30 01:44:03.473887 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 01:44:03.473887 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 01:44:03.473887 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 01:44:03.473887 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 01:44:03.475089 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 01:44:03.475089 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 01:44:03.475089 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 01:44:03.475089 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 01:44:03.475089 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 01:44:03.475089 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 01:44:03.475089 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 01:44:03.475089 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Aug 30 01:44:03.475089 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_service(service_id) Aug 30 01:44:03.475089 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Aug 30 01:44:03.475089 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Aug 30 01:44:03.475089 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 01:44:03.475089 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 01:44:03.475089 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 01:44:03.475089 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 01:44:03.475089 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 01:44:03.475089 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 01:44:03.476008 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 01:44:03.476008 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application with Lock( Aug 30 01:44:03.476008 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 01:44:03.476008 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._enter() Aug 30 01:44:03.476008 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 01:44:03.476008 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 01:44:03.476008 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 01:44:03.476008 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.creator() Aug 30 01:44:03.476008 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 01:44:03.476008 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application created_value = creator( Aug 30 01:44:03.476008 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Aug 30 01:44:03.476008 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 30 01:44:03.476008 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Aug 30 01:44:03.476008 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Aug 30 01:44:03.476008 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 472/938] 10.222.0.27 () {56 vars in 1132 bytes} [Tue Aug 30 01:44:03 2022] GET /identity/v3/services/compute_legacy => generated 95 bytes in 168 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 01:44:03.477093 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a21bc4d-7681-4292-9f8b-28d3fa4d7960 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:44:03.492711 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2476a19-9232-4800-ac4b-b5aabe2d1850 None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:44:03.531537 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ae7d622-4253-4d58-b28d-2cc6ad417e45 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:44:03.532246 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ae7d622-4253-4d58-b28d-2cc6ad417e45 None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:44:03.532446 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ae7d622-4253-4d58-b28d-2cc6ad417e45 None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:44:03.532673 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ae7d622-4253-4d58-b28d-2cc6ad417e45 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:44:03.533367 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ae7d622-4253-4d58-b28d-2cc6ad417e45 None admin] RBAC: Adding query filter params (interface=public, service_id=54dcf8b2a2e74b31b6293369850f6fdf, region_id=RegionOne) {{(pid=81578) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:44:03.533703 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ae7d622-4253-4d58-b28d-2cc6ad417e45 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:44:03.535049 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ae7d622-4253-4d58-b28d-2cc6ad417e45 None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:44:03.545131 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 467/939] 10.222.0.27 () {56 vars in 1251 bytes} [Tue Aug 30 01:44:03 2022] GET /identity/v3/endpoints?service_id=54dcf8b2a2e74b31b6293369850f6fdf&interface=public®ion_id=RegionOne => generated 194 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:44:03.545131 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ae7d622-4253-4d58-b28d-2cc6ad417e45 None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:44:03.584499 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:44:03.591957 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:44:03.596484 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a21bc4d-7681-4292-9f8b-28d3fa4d7960 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:44:03.632697 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d40a0f5-8523-4b6f-a691-60c1145a172d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:44:03.633341 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d40a0f5-8523-4b6f-a691-60c1145a172d None admin] REQUEST_METHOD: `PUT` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:44:03.633625 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d40a0f5-8523-4b6f-a691-60c1145a172d None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:44:03.633901 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d40a0f5-8523-4b6f-a691-60c1145a172d None admin] PATH_INFO: `/v3/projects/4562f3951fbe465280073e60a11e44e2/users/121c374d49a64335ab74a431cfbc1320/roles/8e1192374afc4af6a7cea01c69e72402` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:44:03.636399 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d40a0f5-8523-4b6f-a691-60c1145a172d None admin] RBAC: Authorizing `identity:create_grant(project_id=4562f3951fbe465280073e60a11e44e2, user_id=121c374d49a64335ab74a431cfbc1320, role_id=8e1192374afc4af6a7cea01c69e72402)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:44:03.638437 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d40a0f5-8523-4b6f-a691-60c1145a172d None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:44:03.658120 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ae7d622-4253-4d58-b28d-2cc6ad417e45 None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:44:03.695688 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-474d7425-7641-4c9f-b52b-f3896e78a921 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:44:03.696333 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-474d7425-7641-4c9f-b52b-f3896e78a921 None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:44:03.696572 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-474d7425-7641-4c9f-b52b-f3896e78a921 None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:44:03.696892 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-474d7425-7641-4c9f-b52b-f3896e78a921 None admin] PATH_INFO: `/v3/services` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:44:03.697563 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-474d7425-7641-4c9f-b52b-f3896e78a921 None admin] RBAC: Adding query filter params (name=compute_legacy) {{(pid=81578) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:44:03.697867 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-474d7425-7641-4c9f-b52b-f3896e78a921 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:44:03.699246 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-474d7425-7641-4c9f-b52b-f3896e78a921 None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:44:03.706821 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 468/940] 10.222.0.27 () {56 vars in 1126 bytes} [Tue Aug 30 01:44:03 2022] GET /identity/v3/services?name=compute_legacy => generated 131 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:44:03.708602 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-474d7425-7641-4c9f-b52b-f3896e78a921 None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:44:03.749598 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 473/941] 10.222.0.27 () {58 vars in 1439 bytes} [Tue Aug 30 01:44:03 2022] PUT /identity/v3/projects/4562f3951fbe465280073e60a11e44e2/users/121c374d49a64335ab74a431cfbc1320/roles/8e1192374afc4af6a7cea01c69e72402 => generated 0 bytes in 274 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 01:44:03.758664 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d40a0f5-8523-4b6f-a691-60c1145a172d None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:44:03.839817 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-474d7425-7641-4c9f-b52b-f3896e78a921 None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:44:03.850011 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:44:03.885110 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-303f6f39-0567-47a1-96c5-ec5d67fb6ad0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:44:03.885629 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-303f6f39-0567-47a1-96c5-ec5d67fb6ad0 None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:44:03.886067 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-303f6f39-0567-47a1-96c5-ec5d67fb6ad0 None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:44:03.886533 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-303f6f39-0567-47a1-96c5-ec5d67fb6ad0 None admin] PATH_INFO: `/v3/services` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:44:03.887364 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-303f6f39-0567-47a1-96c5-ec5d67fb6ad0 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:44:03.889071 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-303f6f39-0567-47a1-96c5-ec5d67fb6ad0 None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:44:03.902810 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d40a0f5-8523-4b6f-a691-60c1145a172d None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:44:03.903480 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 469/942] 10.222.0.27 () {56 vars in 1087 bytes} [Tue Aug 30 01:44:03 2022] GET /identity/v3/services => generated 1497 bytes in 196 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 01:44:03.944963 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77533813-e6a1-4e12-a591-1e0b21edd85e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:44:03.945684 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77533813-e6a1-4e12-a591-1e0b21edd85e None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:44:03.945967 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77533813-e6a1-4e12-a591-1e0b21edd85e None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:44:03.946239 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77533813-e6a1-4e12-a591-1e0b21edd85e None admin] PATH_INFO: `/v3/services` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:44:03.946902 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77533813-e6a1-4e12-a591-1e0b21edd85e None admin] RBAC: Adding query filter params (type=compute_legacy) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:44:03.947206 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77533813-e6a1-4e12-a591-1e0b21edd85e None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:44:03.948616 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77533813-e6a1-4e12-a591-1e0b21edd85e None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:44:03.958127 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 474/943] 10.222.0.27 () {56 vars in 1126 bytes} [Tue Aug 30 01:44:03 2022] GET /identity/v3/services?type=compute_legacy => generated 386 bytes in 196 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:44:03.961422 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-303f6f39-0567-47a1-96c5-ec5d67fb6ad0 None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:44:04.075358 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-303f6f39-0567-47a1-96c5-ec5d67fb6ad0 None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:44:04.111422 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8d44ca6-2c04-4782-a627-cbd04a75c630 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:44:04.112028 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8d44ca6-2c04-4782-a627-cbd04a75c630 None admin] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:44:04.112299 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8d44ca6-2c04-4782-a627-cbd04a75c630 None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:44:04.112567 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8d44ca6-2c04-4782-a627-cbd04a75c630 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:44:04.113240 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8d44ca6-2c04-4782-a627-cbd04a75c630 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:44:04.115269 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8d44ca6-2c04-4782-a627-cbd04a75c630 None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:44:04.220416 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 470/944] 10.222.0.27 () {60 vars in 1144 bytes} [Tue Aug 30 01:44:03 2022] POST /identity/v3/endpoints => generated 347 bytes in 259 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 30 01:44:05.174366 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-940b3959-e7b4-4277-92fa-287de9f2a71f None None] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:44:05.174917 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-940b3959-e7b4-4277-92fa-287de9f2a71f None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:44:05.174917 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-940b3959-e7b4-4277-92fa-287de9f2a71f None None] PATH_INFO: `/` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:44:05.176881 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 475/945] 10.222.0.27 () {54 vars in 927 bytes} [Tue Aug 30 01:44:05 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 01:44:05.182224 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d2d236f-6108-48e7-8689-e76d1bb37a37 None None] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:44:05.182998 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d2d236f-6108-48e7-8689-e76d1bb37a37 None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:44:05.183482 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d2d236f-6108-48e7-8689-e76d1bb37a37 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:44:05.202335 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.auth.core [None req-0d2d236f-6108-48e7-8689-e76d1bb37a37 None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81578) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 01:44:05.203870 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-0d2d236f-6108-48e7-8689-e76d1bb37a37 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:44:05.222816 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:44:05.280459 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 471/946] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Aug 30 01:44:05 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Aug 30 01:44:05.555962 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-940b3959-e7b4-4277-92fa-287de9f2a71f None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:44:05.571927 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:44:05.663563 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-940b3959-e7b4-4277-92fa-287de9f2a71f None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:44:05.700342 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1d118a2-ff7f-4ff0-89d2-41b50c732a44 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:44:05.701338 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1d118a2-ff7f-4ff0-89d2-41b50c732a44 None admin] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:44:05.701852 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1d118a2-ff7f-4ff0-89d2-41b50c732a44 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:44:05.702331 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1d118a2-ff7f-4ff0-89d2-41b50c732a44 None admin] PATH_INFO: `/v3/projects` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:44:05.703239 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1d118a2-ff7f-4ff0-89d2-41b50c732a44 None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:44:05.704954 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1d118a2-ff7f-4ff0-89d2-41b50c732a44 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:44:05.708038 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: WARNING keystone.server.flask.common [None req-b1d118a2-ff7f-4ff0-89d2-41b50c732a44 None admin] Deprecated: Not specifying a domain during a create user, group or project call, and relying on falling back to the default domain, is deprecated as of Liberty. There is no plan to remove this compatibility, however, future API versions may remove this, so please specify the domain explicitly or use a domain-scoped token. Aug 30 01:44:05.755860 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.sql.core [None req-b1d118a2-ff7f-4ff0-89d2-41b50c732a44 None admin] Conflict project: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'default-service' for key 'project.ixu_project_name_domain_id'") Aug 30 01:44:05.755860 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [SQL: INSERT INTO project (id, name, domain_id, description, enabled, extra, parent_id, is_domain) VALUES (%(id)s, %(name)s, %(domain_id)s, %(description)s, %(enabled)s, %(extra)s, %(parent_id)s, %(is_domain)s)] Aug 30 01:44:05.755860 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [parameters: {'id': 'ca9f6d0cbe434eeb8894d69ecadae8d2', 'name': 'service', 'domain_id': 'default', 'description': '', 'enabled': 1, 'extra': '{}', 'parent_id': 'default', 'is_domain': 0}] Aug 30 01:44:05.755860 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: (Background on this error at: https://sqlalche.me/e/14/gkpj) {{(pid=81577) wrapper /opt/stack/keystone/keystone/common/sql/core.py:533}} Aug 30 01:44:05.764442 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application [None req-b1d118a2-ff7f-4ff0-89d2-41b50c732a44 None admin] Conflict occurred attempting to store project - it is not permitted to have two projects with either the same name or same id in the same domain: name is service, project id ca9f6d0cbe434eeb8894d69ecadae8d2.: keystone.exception.Conflict: Conflict occurred attempting to store project - it is not permitted to have two projects with either the same name or same id in the same domain: name is service, project id ca9f6d0cbe434eeb8894d69ecadae8d2. Aug 30 01:44:05.764442 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 01:44:05.764442 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Aug 30 01:44:05.764442 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application self.dialect.do_execute( Aug 30 01:44:05.764442 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Aug 30 01:44:05.764442 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Aug 30 01:44:05.764442 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Aug 30 01:44:05.764442 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application result = self._query(query) Aug 30 01:44:05.764442 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Aug 30 01:44:05.764442 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application conn.query(q) Aug 30 01:44:05.764442 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Aug 30 01:44:05.764442 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Aug 30 01:44:05.764442 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Aug 30 01:44:05.765285 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application result.read() Aug 30 01:44:05.765285 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Aug 30 01:44:05.765285 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Aug 30 01:44:05.765285 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Aug 30 01:44:05.765285 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application packet.raise_for_error() Aug 30 01:44:05.765285 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Aug 30 01:44:05.765285 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Aug 30 01:44:05.765285 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Aug 30 01:44:05.765285 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Aug 30 01:44:05.765285 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application pymysql.err.IntegrityError: (1062, "Duplicate entry 'default-service' for key 'project.ixu_project_name_domain_id'") Aug 30 01:44:05.765285 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Aug 30 01:44:05.765285 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application The above exception was the direct cause of the following exception: Aug 30 01:44:05.765285 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Aug 30 01:44:05.765285 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 01:44:05.765285 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 528, in wrapper Aug 30 01:44:05.765285 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return method(*args, **kwargs) Aug 30 01:44:05.765285 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 241, in create_project Aug 30 01:44:05.765285 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return project_ref.to_dict() Aug 30 01:44:05.766355 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Aug 30 01:44:05.766355 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application next(self.gen) Aug 30 01:44:05.766355 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 1060, in _transaction_scope Aug 30 01:44:05.766355 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application yield resource Aug 30 01:44:05.766355 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Aug 30 01:44:05.766355 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application next(self.gen) Aug 30 01:44:05.766355 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 662, in _session Aug 30 01:44:05.766355 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application self.session.rollback() Aug 30 01:44:05.766355 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Aug 30 01:44:05.766355 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application self.force_reraise() Aug 30 01:44:05.766355 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Aug 30 01:44:05.766355 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise self.value Aug 30 01:44:05.766355 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 659, in _session Aug 30 01:44:05.766355 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application self._end_session_transaction(self.session) Aug 30 01:44:05.766355 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 687, in _end_session_transaction Aug 30 01:44:05.766355 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application session.commit() Aug 30 01:44:05.766355 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 1451, in commit Aug 30 01:44:05.767268 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application self._transaction.commit(_to_root=self.future) Aug 30 01:44:05.767268 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 829, in commit Aug 30 01:44:05.767268 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application self._prepare_impl() Aug 30 01:44:05.767268 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 808, in _prepare_impl Aug 30 01:44:05.767268 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application self.session.flush() Aug 30 01:44:05.767268 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3383, in flush Aug 30 01:44:05.767268 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application self._flush(objects) Aug 30 01:44:05.767268 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3523, in _flush Aug 30 01:44:05.767268 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application transaction.rollback(_capture_exception=True) Aug 30 01:44:05.767268 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__ Aug 30 01:44:05.767268 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application compat.raise_( Aug 30 01:44:05.767268 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Aug 30 01:44:05.767268 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise exception Aug 30 01:44:05.767268 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3483, in _flush Aug 30 01:44:05.767268 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application flush_context.execute() Aug 30 01:44:05.767268 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 456, in execute Aug 30 01:44:05.767268 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application rec.execute(self) Aug 30 01:44:05.768186 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 630, in execute Aug 30 01:44:05.768186 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application util.preloaded.orm_persistence.save_obj( Aug 30 01:44:05.768186 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 245, in save_obj Aug 30 01:44:05.768186 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application _emit_insert_statements( Aug 30 01:44:05.768186 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 1097, in _emit_insert_statements Aug 30 01:44:05.768186 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application c = connection._execute_20( Aug 30 01:44:05.768186 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1705, in _execute_20 Aug 30 01:44:05.768186 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return meth(self, args_10style, kwargs_10style, execution_options) Aug 30 01:44:05.768186 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/sql/elements.py", line 333, in _execute_on_connection Aug 30 01:44:05.768186 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return connection._execute_clauseelement( Aug 30 01:44:05.768186 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1572, in _execute_clauseelement Aug 30 01:44:05.768186 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application ret = self._execute_context( Aug 30 01:44:05.768186 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1943, in _execute_context Aug 30 01:44:05.768186 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application self._handle_dbapi_exception( Aug 30 01:44:05.768186 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 2122, in _handle_dbapi_exception Aug 30 01:44:05.768983 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application util.raise_(newraise, with_traceback=exc_info[2], from_=e) Aug 30 01:44:05.768983 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Aug 30 01:44:05.768983 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise exception Aug 30 01:44:05.768983 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Aug 30 01:44:05.768983 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application self.dialect.do_execute( Aug 30 01:44:05.768983 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Aug 30 01:44:05.768983 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Aug 30 01:44:05.768983 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Aug 30 01:44:05.768983 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application result = self._query(query) Aug 30 01:44:05.768983 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Aug 30 01:44:05.768983 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application conn.query(q) Aug 30 01:44:05.768983 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Aug 30 01:44:05.768983 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Aug 30 01:44:05.768983 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Aug 30 01:44:05.768983 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application result.read() Aug 30 01:44:05.768983 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Aug 30 01:44:05.768983 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Aug 30 01:44:05.771852 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Aug 30 01:44:05.771852 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application packet.raise_for_error() Aug 30 01:44:05.771852 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Aug 30 01:44:05.771852 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Aug 30 01:44:05.771852 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Aug 30 01:44:05.771852 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Aug 30 01:44:05.771852 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application oslo_db.exception.DBDuplicateEntry: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'default-service' for key 'project.ixu_project_name_domain_id'") Aug 30 01:44:05.771852 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application [SQL: INSERT INTO project (id, name, domain_id, description, enabled, extra, parent_id, is_domain) VALUES (%(id)s, %(name)s, %(domain_id)s, %(description)s, %(enabled)s, %(extra)s, %(parent_id)s, %(is_domain)s)] Aug 30 01:44:05.771852 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application [parameters: {'id': 'ca9f6d0cbe434eeb8894d69ecadae8d2', 'name': 'service', 'domain_id': 'default', 'description': '', 'enabled': 1, 'extra': '{}', 'parent_id': 'default', 'is_domain': 0}] Aug 30 01:44:05.771852 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application (Background on this error at: https://sqlalche.me/e/14/gkpj) Aug 30 01:44:05.771852 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Aug 30 01:44:05.771852 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 30 01:44:05.771852 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Aug 30 01:44:05.771852 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 01:44:05.771852 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 216, in create_project Aug 30 01:44:05.773872 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application ret = self.driver.create_project(project_id, project) Aug 30 01:44:05.773872 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 563, in wrapper Aug 30 01:44:05.773872 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise exception.Conflict(type=conflict_type, Aug 30 01:44:05.773872 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application keystone.exception.Conflict: Conflict occurred attempting to store project - Duplicate entry found with name service at domain ID default. Aug 30 01:44:05.773872 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Aug 30 01:44:05.773872 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 30 01:44:05.773872 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Aug 30 01:44:05.773872 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 01:44:05.773872 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 01:44:05.773872 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 01:44:05.773872 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 01:44:05.773872 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 01:44:05.773872 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 01:44:05.773872 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 01:44:05.773872 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 01:44:05.773872 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 01:44:05.773872 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 01:44:05.776097 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 01:44:05.776097 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 176, in post Aug 30 01:44:05.776097 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application ref = PROVIDERS.resource_api.create_project( Aug 30 01:44:05.776097 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 01:44:05.776097 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 01:44:05.776097 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 218, in create_project Aug 30 01:44:05.776097 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise exception.Conflict( Aug 30 01:44:05.776097 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application keystone.exception.Conflict: Conflict occurred attempting to store project - it is not permitted to have two projects with either the same name or same id in the same domain: name is service, project id ca9f6d0cbe434eeb8894d69ecadae8d2. Aug 30 01:44:05.776097 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Aug 30 01:44:05.779172 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 476/947] 10.222.0.27 () {60 vars in 1140 bytes} [Tue Aug 30 01:44:05 2022] POST /identity/v3/projects => generated 262 bytes in 224 msecs (HTTP/1.1 409) 5 headers in 183 bytes (1 switches on core 0) Aug 30 01:44:05.786862 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d2d236f-6108-48e7-8689-e76d1bb37a37 None None] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:44:05.882691 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-599233c8-832b-4e77-9a50-26c3b54f4d06 None None] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:44:05.883465 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-599233c8-832b-4e77-9a50-26c3b54f4d06 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:44:05.884001 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-599233c8-832b-4e77-9a50-26c3b54f4d06 None None] PATH_INFO: `/` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:44:05.889292 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 477/948] 10.222.0.27 () {54 vars in 927 bytes} [Tue Aug 30 01:44:05 2022] GET /identity => generated 270 bytes in 8 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 30 01:44:05.893551 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d2d236f-6108-48e7-8689-e76d1bb37a37 None None] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:44:05.895787 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c661580b-3f7a-4a9b-8107-071ad5e3cbd3 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:44:05.896432 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c661580b-3f7a-4a9b-8107-071ad5e3cbd3 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:44:05.896923 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c661580b-3f7a-4a9b-8107-071ad5e3cbd3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:44:05.918875 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-c661580b-3f7a-4a9b-8107-071ad5e3cbd3 None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 01:44:05.920391 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-c661580b-3f7a-4a9b-8107-071ad5e3cbd3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:44:05.929986 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ae83319-6409-4de6-9e4a-c0a84592596c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:44:05.930614 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ae83319-6409-4de6-9e4a-c0a84592596c None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:44:05.931115 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ae83319-6409-4de6-9e4a-c0a84592596c None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:44:05.931225 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ae83319-6409-4de6-9e4a-c0a84592596c None admin] PATH_INFO: `/v3/projects/service` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:44:05.938593 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ae83319-6409-4de6-9e4a-c0a84592596c None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:44:05.940017 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ae83319-6409-4de6-9e4a-c0a84592596c None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:44:05.952506 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application [None req-1ae83319-6409-4de6-9e4a-c0a84592596c None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 30 01:44:05.952506 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 01:44:05.952506 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 01:44:05.952506 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 01:44:05.952506 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 01:44:05.952506 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 01:44:05.952506 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 01:44:05.952506 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 01:44:05.952506 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 01:44:05.952506 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 01:44:05.952506 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 01:44:05.952506 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 01:44:05.952506 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 30 01:44:05.952506 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 30 01:44:05.952506 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 30 01:44:05.953536 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 30 01:44:05.953536 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 01:44:05.953536 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 01:44:05.953536 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 01:44:05.953536 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 01:44:05.953536 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 01:44:05.953536 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 01:44:05.953536 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 01:44:05.953536 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application with Lock( Aug 30 01:44:05.953536 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 01:44:05.953536 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._enter() Aug 30 01:44:05.953536 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 01:44:05.953536 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 01:44:05.953536 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 01:44:05.953536 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.creator() Aug 30 01:44:05.953536 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 01:44:05.953536 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application created_value = creator( Aug 30 01:44:05.960626 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 30 01:44:05.960626 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 30 01:44:05.960626 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 30 01:44:05.960626 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 30 01:44:05.960626 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 30 01:44:05.960626 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 30 01:44:05.960626 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 30 01:44:05.960626 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application Aug 30 01:44:05.960626 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 472/949] 10.222.0.27 () {56 vars in 1111 bytes} [Tue Aug 30 01:44:05 2022] GET /identity/v3/projects/service => generated 88 bytes in 169 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 01:44:05.960626 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60cf5c8c-b9b9-4cf3-8ba0-c4daf1e228aa None None] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:44:05.960626 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60cf5c8c-b9b9-4cf3-8ba0-c4daf1e228aa None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:44:05.961356 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60cf5c8c-b9b9-4cf3-8ba0-c4daf1e228aa None None] PATH_INFO: `/` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:44:05.961356 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 473/950] 10.222.0.27 () {54 vars in 927 bytes} [Tue Aug 30 01:44:05 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 01:44:05.961356 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60cf5c8c-b9b9-4cf3-8ba0-c4daf1e228aa None None] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:44:05.973240 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:44:05.998029 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:44:06.003706 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 478/951] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Aug 30 01:44:05 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 30 01:44:06.005600 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4cc1657-4ca1-4301-916f-3310d1f30e4f None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:44:06.006086 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4cc1657-4ca1-4301-916f-3310d1f30e4f None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:44:06.006493 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4cc1657-4ca1-4301-916f-3310d1f30e4f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:44:06.030962 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-c4cc1657-4ca1-4301-916f-3310d1f30e4f None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 01:44:06.032649 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-c4cc1657-4ca1-4301-916f-3310d1f30e4f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:44:06.084758 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60cf5c8c-b9b9-4cf3-8ba0-c4daf1e228aa None None] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:44:06.109873 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 479/952] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Aug 30 01:44:06 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 30 01:44:06.126206 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6174ce8e-217d-4911-82a4-cebf23b665dc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:44:06.127037 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6174ce8e-217d-4911-82a4-cebf23b665dc None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:44:06.127475 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6174ce8e-217d-4911-82a4-cebf23b665dc None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:44:06.127904 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6174ce8e-217d-4911-82a4-cebf23b665dc None admin] PATH_INFO: `/v3/projects` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:44:06.128705 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6174ce8e-217d-4911-82a4-cebf23b665dc None admin] RBAC: Adding query filter params (name=service) {{(pid=81578) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:44:06.129155 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6174ce8e-217d-4911-82a4-cebf23b665dc None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:44:06.130907 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6174ce8e-217d-4911-82a4-cebf23b665dc None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:44:06.149115 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 474/953] 10.222.0.27 () {56 vars in 1112 bytes} [Tue Aug 30 01:44:05 2022] GET /identity/v3/projects?name=service => generated 411 bytes in 189 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:44:06.193236 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c066e672-2944-4a32-a6cf-a8e79f55b00a None None] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:44:06.194236 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c066e672-2944-4a32-a6cf-a8e79f55b00a None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:44:06.194708 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c066e672-2944-4a32-a6cf-a8e79f55b00a None None] PATH_INFO: `/` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:44:06.196670 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 480/954] 10.222.0.27 () {54 vars in 927 bytes} [Tue Aug 30 01:44:06 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 01:44:06.202529 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f8154ef-5aa9-4a6f-87d0-02a9cb4b2e71 None None] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:44:06.203073 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f8154ef-5aa9-4a6f-87d0-02a9cb4b2e71 None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:44:06.203073 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f8154ef-5aa9-4a6f-87d0-02a9cb4b2e71 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:44:06.225616 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.auth.core [None req-3f8154ef-5aa9-4a6f-87d0-02a9cb4b2e71 None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81578) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 01:44:06.227161 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-3f8154ef-5aa9-4a6f-87d0-02a9cb4b2e71 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:44:06.303458 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c066e672-2944-4a32-a6cf-a8e79f55b00a None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:44:06.305181 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:44:06.314276 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 475/955] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Aug 30 01:44:06 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 112 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 30 01:44:06.392542 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f8154ef-5aa9-4a6f-87d0-02a9cb4b2e71 None None] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:44:06.420183 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c066e672-2944-4a32-a6cf-a8e79f55b00a None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:44:06.454348 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:44:06.466666 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43e83ca0-2386-44f9-94bd-685fea06fbf6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:44:06.467402 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43e83ca0-2386-44f9-94bd-685fea06fbf6 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:44:06.467921 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43e83ca0-2386-44f9-94bd-685fea06fbf6 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:44:06.468519 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43e83ca0-2386-44f9-94bd-685fea06fbf6 None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:44:06.487213 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43e83ca0-2386-44f9-94bd-685fea06fbf6 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 30 01:44:06.492961 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application [None req-43e83ca0-2386-44f9-94bd-685fea06fbf6 None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Aug 30 01:44:06.492961 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 01:44:06.492961 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 01:44:06.492961 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 01:44:06.492961 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 01:44:06.492961 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 01:44:06.492961 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 01:44:06.492961 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 01:44:06.492961 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 01:44:06.492961 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 01:44:06.492961 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 01:44:06.492961 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 01:44:06.492961 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 30 01:44:06.492961 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 30 01:44:06.492961 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 30 01:44:06.492961 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 30 01:44:06.493991 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 30 01:44:06.493991 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 30 01:44:06.493991 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 30 01:44:06.493991 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 30 01:44:06.493991 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 01:44:06.493991 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 01:44:06.493991 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 01:44:06.493991 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 01:44:06.493991 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 01:44:06.493991 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 01:44:06.493991 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 01:44:06.493991 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application with Lock( Aug 30 01:44:06.493991 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 01:44:06.493991 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._enter() Aug 30 01:44:06.493991 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 01:44:06.493991 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 01:44:06.494917 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 01:44:06.494917 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.creator() Aug 30 01:44:06.494917 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 01:44:06.494917 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application created_value = creator( Aug 30 01:44:06.494917 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 30 01:44:06.494917 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 30 01:44:06.494917 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 30 01:44:06.494917 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 30 01:44:06.494917 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 30 01:44:06.494917 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 30 01:44:06.494917 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Aug 30 01:44:06.494917 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Aug 30 01:44:06.494917 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 481/956] 10.222.0.27 () {56 vars in 1120 bytes} [Tue Aug 30 01:44:06 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 192 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 01:44:06.501734 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43e83ca0-2386-44f9-94bd-685fea06fbf6 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:44:06.521599 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f8154ef-5aa9-4a6f-87d0-02a9cb4b2e71 None None] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:44:06.565944 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f2c291d-9a02-4f53-b6e9-2194d2541a27 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:44:06.566684 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f2c291d-9a02-4f53-b6e9-2194d2541a27 None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:44:06.567595 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f2c291d-9a02-4f53-b6e9-2194d2541a27 None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:44:06.567902 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f2c291d-9a02-4f53-b6e9-2194d2541a27 None admin] PATH_INFO: `/v3/services/object-store` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:44:06.568584 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f2c291d-9a02-4f53-b6e9-2194d2541a27 None admin] RBAC: Authorizing `identity:get_service(service_id=object-store)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:44:06.570042 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f2c291d-9a02-4f53-b6e9-2194d2541a27 None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:44:06.579487 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application [None req-6f2c291d-9a02-4f53-b6e9-2194d2541a27 None admin] Could not find service: object-store.: keystone.exception.ServiceNotFound: Could not find service: object-store. Aug 30 01:44:06.579487 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 01:44:06.579487 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Aug 30 01:44:06.579487 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Aug 30 01:44:06.579487 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Aug 30 01:44:06.579487 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Aug 30 01:44:06.579487 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Aug 30 01:44:06.579487 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 30 01:44:06.579487 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Aug 30 01:44:06.579487 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application Aug 30 01:44:06.579487 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 30 01:44:06.579487 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application Aug 30 01:44:06.579487 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 01:44:06.579487 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 01:44:06.579487 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 01:44:06.579487 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 01:44:06.579487 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 01:44:06.580624 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 01:44:06.580624 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 01:44:06.580624 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 01:44:06.580624 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 01:44:06.580624 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 01:44:06.580624 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 01:44:06.580624 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Aug 30 01:44:06.580624 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._get_service(service_id) Aug 30 01:44:06.580624 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Aug 30 01:44:06.580624 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Aug 30 01:44:06.580624 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 01:44:06.580624 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 01:44:06.580624 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 01:44:06.580624 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 01:44:06.580624 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 01:44:06.580624 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 01:44:06.581503 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 01:44:06.581503 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application with Lock( Aug 30 01:44:06.581503 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 01:44:06.581503 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._enter() Aug 30 01:44:06.581503 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 01:44:06.581503 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 01:44:06.581503 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 01:44:06.581503 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.creator() Aug 30 01:44:06.581503 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 01:44:06.581503 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application created_value = creator( Aug 30 01:44:06.581503 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Aug 30 01:44:06.581503 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 30 01:44:06.581503 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Aug 30 01:44:06.581503 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application Aug 30 01:44:06.581503 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 476/957] 10.222.0.27 () {56 vars in 1126 bytes} [Tue Aug 30 01:44:06 2022] GET /identity/v3/services/object-store => generated 93 bytes in 189 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 30 01:44:06.586401 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f2c291d-9a02-4f53-b6e9-2194d2541a27 None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:44:06.626015 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43e83ca0-2386-44f9-94bd-685fea06fbf6 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:44:06.658610 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:44:06.665435 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb4c7684-d271-4cf2-8afc-be0f316464d5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:44:06.666156 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb4c7684-d271-4cf2-8afc-be0f316464d5 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:44:06.666510 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb4c7684-d271-4cf2-8afc-be0f316464d5 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:44:06.666914 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb4c7684-d271-4cf2-8afc-be0f316464d5 None admin] PATH_INFO: `/v3/roles` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:44:06.667666 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb4c7684-d271-4cf2-8afc-be0f316464d5 None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:44:06.668050 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb4c7684-d271-4cf2-8afc-be0f316464d5 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:44:06.669563 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb4c7684-d271-4cf2-8afc-be0f316464d5 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:44:06.678842 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 482/958] 10.222.0.27 () {56 vars in 1115 bytes} [Tue Aug 30 01:44:06 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 30 01:44:06.679567 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb4c7684-d271-4cf2-8afc-be0f316464d5 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:44:06.706731 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f2c291d-9a02-4f53-b6e9-2194d2541a27 None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:44:06.744049 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:44:06.748873 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e0e6111-e1b4-4ce8-b66d-787df940d00c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:44:06.749498 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e0e6111-e1b4-4ce8-b66d-787df940d00c None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:44:06.749823 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e0e6111-e1b4-4ce8-b66d-787df940d00c None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:44:06.750181 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e0e6111-e1b4-4ce8-b66d-787df940d00c None admin] PATH_INFO: `/v3/services` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:44:06.756305 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e0e6111-e1b4-4ce8-b66d-787df940d00c None admin] RBAC: Adding query filter params (name=object-store) {{(pid=81578) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:44:06.756305 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e0e6111-e1b4-4ce8-b66d-787df940d00c None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:44:06.756305 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e0e6111-e1b4-4ce8-b66d-787df940d00c None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:44:06.761616 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 477/959] 10.222.0.27 () {56 vars in 1122 bytes} [Tue Aug 30 01:44:06 2022] GET /identity/v3/services?name=object-store => generated 129 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:44:06.762579 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e0e6111-e1b4-4ce8-b66d-787df940d00c None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:44:06.812739 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb4c7684-d271-4cf2-8afc-be0f316464d5 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:44:06.849802 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-897df909-f9ab-45e0-83de-590bd24da5df None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:44:06.850438 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-897df909-f9ab-45e0-83de-590bd24da5df None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:44:06.850721 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-897df909-f9ab-45e0-83de-590bd24da5df None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:44:06.850998 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-897df909-f9ab-45e0-83de-590bd24da5df None admin] PATH_INFO: `/v3/services/compute` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:44:06.851843 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-897df909-f9ab-45e0-83de-590bd24da5df None admin] RBAC: Authorizing `identity:get_service(service_id=compute)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:44:06.853208 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-897df909-f9ab-45e0-83de-590bd24da5df None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:44:06.862132 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application [None req-897df909-f9ab-45e0-83de-590bd24da5df None admin] Could not find service: compute.: keystone.exception.ServiceNotFound: Could not find service: compute. Aug 30 01:44:06.862132 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 01:44:06.862132 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Aug 30 01:44:06.862132 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Aug 30 01:44:06.862132 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Aug 30 01:44:06.862132 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Aug 30 01:44:06.862132 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Aug 30 01:44:06.862132 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 30 01:44:06.862132 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Aug 30 01:44:06.862132 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Aug 30 01:44:06.862132 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 30 01:44:06.862132 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Aug 30 01:44:06.862132 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 01:44:06.862132 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 01:44:06.862132 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 01:44:06.862132 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 01:44:06.862132 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 01:44:06.869485 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 01:44:06.869485 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 01:44:06.869485 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 01:44:06.869485 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 01:44:06.869485 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 01:44:06.869485 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 01:44:06.869485 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Aug 30 01:44:06.869485 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_service(service_id) Aug 30 01:44:06.869485 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Aug 30 01:44:06.869485 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Aug 30 01:44:06.869485 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 01:44:06.869485 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 01:44:06.869485 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 01:44:06.869485 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 01:44:06.869485 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 01:44:06.869485 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 01:44:06.870531 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 01:44:06.870531 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application with Lock( Aug 30 01:44:06.870531 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 01:44:06.870531 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._enter() Aug 30 01:44:06.870531 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 01:44:06.870531 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 01:44:06.870531 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 01:44:06.870531 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.creator() Aug 30 01:44:06.870531 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 01:44:06.870531 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application created_value = creator( Aug 30 01:44:06.870531 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Aug 30 01:44:06.870531 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 30 01:44:06.870531 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Aug 30 01:44:06.870531 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Aug 30 01:44:06.870531 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 483/960] 10.222.0.27 () {56 vars in 1111 bytes} [Tue Aug 30 01:44:06 2022] GET /identity/v3/services/compute => generated 88 bytes in 184 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 01:44:06.871678 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-897df909-f9ab-45e0-83de-590bd24da5df None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:44:06.885962 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e0e6111-e1b4-4ce8-b66d-787df940d00c None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:44:06.915393 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:44:06.925484 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0794ecf-f22e-419e-b17c-2edd4a4b05f3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:44:06.929869 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0794ecf-f22e-419e-b17c-2edd4a4b05f3 None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:44:06.929869 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0794ecf-f22e-419e-b17c-2edd4a4b05f3 None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:44:06.929869 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0794ecf-f22e-419e-b17c-2edd4a4b05f3 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:44:06.938106 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0794ecf-f22e-419e-b17c-2edd4a4b05f3 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:44:06.939534 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0794ecf-f22e-419e-b17c-2edd4a4b05f3 None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:44:06.956202 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application [None req-d0794ecf-f22e-419e-b17c-2edd4a4b05f3 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 01:44:06.956202 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 01:44:06.956202 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 01:44:06.956202 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 01:44:06.956202 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 01:44:06.956202 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 01:44:06.956202 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 01:44:06.956202 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 01:44:06.956202 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 01:44:06.956202 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 01:44:06.956202 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 01:44:06.956202 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 01:44:06.956202 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 30 01:44:06.956202 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 30 01:44:06.956202 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 30 01:44:06.956202 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 30 01:44:06.957892 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 01:44:06.957892 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 01:44:06.957892 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 01:44:06.957892 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 01:44:06.957892 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 01:44:06.957892 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 01:44:06.957892 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 01:44:06.957892 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application with Lock( Aug 30 01:44:06.957892 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 01:44:06.957892 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._enter() Aug 30 01:44:06.957892 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 01:44:06.957892 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 01:44:06.957892 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 01:44:06.957892 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.creator() Aug 30 01:44:06.957892 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 01:44:06.957892 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application created_value = creator( Aug 30 01:44:06.957892 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 30 01:44:06.963983 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 30 01:44:06.963983 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 01:44:06.963983 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application Aug 30 01:44:06.963983 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 478/961] 10.222.0.27 () {56 vars in 1108 bytes} [Tue Aug 30 01:44:06 2022] GET /identity/v3/domains/Default => generated 87 bytes in 196 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 01:44:06.963983 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0794ecf-f22e-419e-b17c-2edd4a4b05f3 None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:44:06.983651 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-897df909-f9ab-45e0-83de-590bd24da5df None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:44:06.997779 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:44:07.021013 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8262fc2b-bd63-4632-8646-e51b25282ecb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:44:07.021728 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8262fc2b-bd63-4632-8646-e51b25282ecb None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:44:07.022076 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8262fc2b-bd63-4632-8646-e51b25282ecb None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:44:07.022416 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8262fc2b-bd63-4632-8646-e51b25282ecb None admin] PATH_INFO: `/v3/services` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:44:07.023115 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8262fc2b-bd63-4632-8646-e51b25282ecb None admin] RBAC: Adding query filter params (type=object-store) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:44:07.023486 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8262fc2b-bd63-4632-8646-e51b25282ecb None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:44:07.024969 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8262fc2b-bd63-4632-8646-e51b25282ecb None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:44:07.029978 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 484/962] 10.222.0.27 () {56 vars in 1122 bytes} [Tue Aug 30 01:44:06 2022] GET /identity/v3/services?type=object-store => generated 356 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:44:07.030639 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8262fc2b-bd63-4632-8646-e51b25282ecb None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:44:07.079912 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0794ecf-f22e-419e-b17c-2edd4a4b05f3 None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:44:07.116038 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8656fb1-e163-4239-82c3-e17e2e3b4ed4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:44:07.116892 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8656fb1-e163-4239-82c3-e17e2e3b4ed4 None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:44:07.117356 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8656fb1-e163-4239-82c3-e17e2e3b4ed4 None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:44:07.117662 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8656fb1-e163-4239-82c3-e17e2e3b4ed4 None admin] PATH_INFO: `/v3/services` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:44:07.118300 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8656fb1-e163-4239-82c3-e17e2e3b4ed4 None admin] RBAC: Adding query filter params (name=compute) {{(pid=81578) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:44:07.118606 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8656fb1-e163-4239-82c3-e17e2e3b4ed4 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:44:07.119960 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8656fb1-e163-4239-82c3-e17e2e3b4ed4 None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:44:07.130104 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 479/963] 10.222.0.27 () {56 vars in 1112 bytes} [Tue Aug 30 01:44:06 2022] GET /identity/v3/services?name=compute => generated 124 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:44:07.130104 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8656fb1-e163-4239-82c3-e17e2e3b4ed4 None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:44:07.143076 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8262fc2b-bd63-4632-8646-e51b25282ecb None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:44:07.180233 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:44:07.180418 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1365ec5d-b84a-40fb-ab77-a80c67019937 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:44:07.180988 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1365ec5d-b84a-40fb-ab77-a80c67019937 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:44:07.181505 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1365ec5d-b84a-40fb-ab77-a80c67019937 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:44:07.181644 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1365ec5d-b84a-40fb-ab77-a80c67019937 None admin] PATH_INFO: `/v3/domains` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:44:07.182195 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1365ec5d-b84a-40fb-ab77-a80c67019937 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:44:07.182498 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1365ec5d-b84a-40fb-ab77-a80c67019937 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:44:07.183894 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1365ec5d-b84a-40fb-ab77-a80c67019937 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:44:07.197064 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 485/964] 10.222.0.27 () {56 vars in 1109 bytes} [Tue Aug 30 01:44:07 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:44:07.197724 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1365ec5d-b84a-40fb-ab77-a80c67019937 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:44:07.245255 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:44:07.246365 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8656fb1-e163-4239-82c3-e17e2e3b4ed4 None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:44:07.291163 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d31b054b-0701-4cfb-8408-ecedc746ab7d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:44:07.292161 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d31b054b-0701-4cfb-8408-ecedc746ab7d None admin] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:44:07.292608 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d31b054b-0701-4cfb-8408-ecedc746ab7d None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:44:07.293021 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d31b054b-0701-4cfb-8408-ecedc746ab7d None admin] PATH_INFO: `/v3/endpoints` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:44:07.293862 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d31b054b-0701-4cfb-8408-ecedc746ab7d None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:44:07.295371 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d31b054b-0701-4cfb-8408-ecedc746ab7d None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:44:07.311810 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1365ec5d-b84a-40fb-ab77-a80c67019937 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:44:07.346930 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a13e6802-673d-41b9-b9e9-d1387bbf8463 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:44:07.347538 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a13e6802-673d-41b9-b9e9-d1387bbf8463 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:44:07.347819 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a13e6802-673d-41b9-b9e9-d1387bbf8463 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:44:07.348203 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a13e6802-673d-41b9-b9e9-d1387bbf8463 None admin] PATH_INFO: `/v3/services` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:44:07.349117 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a13e6802-673d-41b9-b9e9-d1387bbf8463 None admin] RBAC: Adding query filter params (type=compute) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:44:07.349425 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a13e6802-673d-41b9-b9e9-d1387bbf8463 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:44:07.350829 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a13e6802-673d-41b9-b9e9-d1387bbf8463 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:44:07.355476 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 486/965] 10.222.0.27 () {56 vars in 1112 bytes} [Tue Aug 30 01:44:07 2022] GET /identity/v3/services?type=compute => generated 124 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:44:07.356256 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a13e6802-673d-41b9-b9e9-d1387bbf8463 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:44:07.386426 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 480/966] 10.222.0.27 () {60 vars in 1144 bytes} [Tue Aug 30 01:44:07 2022] POST /identity/v3/endpoints => generated 349 bytes in 261 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 01:44:07.487581 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a13e6802-673d-41b9-b9e9-d1387bbf8463 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:44:07.489314 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:44:07.539361 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7531ead-54fa-46dd-bd92-d014fb849685 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:44:07.540005 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7531ead-54fa-46dd-bd92-d014fb849685 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:44:07.540282 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7531ead-54fa-46dd-bd92-d014fb849685 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:44:07.540550 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7531ead-54fa-46dd-bd92-d014fb849685 None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:44:07.550105 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7531ead-54fa-46dd-bd92-d014fb849685 None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:44:07.551704 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7531ead-54fa-46dd-bd92-d014fb849685 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:44:07.564885 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application [None req-a7531ead-54fa-46dd-bd92-d014fb849685 None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Aug 30 01:44:07.564885 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 01:44:07.564885 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 01:44:07.564885 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 01:44:07.564885 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 01:44:07.564885 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 01:44:07.564885 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 01:44:07.564885 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 01:44:07.564885 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 01:44:07.564885 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 01:44:07.564885 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 01:44:07.564885 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 01:44:07.564885 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 30 01:44:07.564885 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 30 01:44:07.564885 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 30 01:44:07.564885 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 30 01:44:07.566039 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 01:44:07.566039 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 01:44:07.566039 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 30 01:44:07.566039 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 30 01:44:07.566039 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 30 01:44:07.566039 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 30 01:44:07.566039 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 01:44:07.566039 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 01:44:07.566039 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 01:44:07.566039 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 01:44:07.566039 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 01:44:07.566039 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application with Lock( Aug 30 01:44:07.566039 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 01:44:07.566039 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._enter() Aug 30 01:44:07.566039 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 01:44:07.566039 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 01:44:07.566039 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 01:44:07.566039 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.creator() Aug 30 01:44:07.568732 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 01:44:07.568732 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application created_value = creator( Aug 30 01:44:07.568732 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 30 01:44:07.568732 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 30 01:44:07.568732 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 30 01:44:07.568732 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 30 01:44:07.568732 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 30 01:44:07.568732 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 30 01:44:07.568732 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Aug 30 01:44:07.568732 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Aug 30 01:44:07.568732 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 487/967] 10.222.0.27 () {56 vars in 1117 bytes} [Tue Aug 30 01:44:07 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 210 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 01:44:07.572477 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d31b054b-0701-4cfb-8408-ecedc746ab7d None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:44:07.680219 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:44:07.684299 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d31b054b-0701-4cfb-8408-ecedc746ab7d None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:44:07.728516 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8251940c-c442-41ba-8663-f625b3fa5892 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:44:07.729256 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8251940c-c442-41ba-8663-f625b3fa5892 None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:44:07.729567 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8251940c-c442-41ba-8663-f625b3fa5892 None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:44:07.729840 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8251940c-c442-41ba-8663-f625b3fa5892 None admin] PATH_INFO: `/v3/users` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:44:07.730583 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8251940c-c442-41ba-8663-f625b3fa5892 None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=81578) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:44:07.731044 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8251940c-c442-41ba-8663-f625b3fa5892 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:44:07.732529 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8251940c-c442-41ba-8663-f625b3fa5892 None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:44:07.752126 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 481/968] 10.222.0.27 () {56 vars in 1149 bytes} [Tue Aug 30 01:44:07 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 180 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:44:07.758331 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7531ead-54fa-46dd-bd92-d014fb849685 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:44:07.867112 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7531ead-54fa-46dd-bd92-d014fb849685 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:44:07.903255 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36aa7a98-5464-48b6-8f2c-90f43cde8170 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:44:07.903979 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36aa7a98-5464-48b6-8f2c-90f43cde8170 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:44:07.904330 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36aa7a98-5464-48b6-8f2c-90f43cde8170 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:44:07.904672 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36aa7a98-5464-48b6-8f2c-90f43cde8170 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:44:07.935483 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:44:07.980087 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36aa7a98-5464-48b6-8f2c-90f43cde8170 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:44:07.981806 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36aa7a98-5464-48b6-8f2c-90f43cde8170 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:44:08.087053 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 488/969] 10.222.0.27 () {58 vars in 1282 bytes} [Tue Aug 30 01:44:07 2022] GET /identity/v3/auth/tokens => generated 1474 bytes in 329 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Aug 30 01:44:08.092845 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8251940c-c442-41ba-8663-f625b3fa5892 None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:44:08.202029 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8251940c-c442-41ba-8663-f625b3fa5892 None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:44:08.226907 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69f7205d-60cc-44a5-aef3-ea03b640141b None None] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:44:08.227404 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69f7205d-60cc-44a5-aef3-ea03b640141b None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:44:08.227632 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69f7205d-60cc-44a5-aef3-ea03b640141b None None] PATH_INFO: `/` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:44:08.229998 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 489/970] 10.222.0.27 () {54 vars in 927 bytes} [Tue Aug 30 01:44:08 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 01:44:08.234983 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8be3bfef-ad67-40a2-a313-0b154a63945d None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:44:08.235283 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8be3bfef-ad67-40a2-a313-0b154a63945d None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:44:08.235558 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8be3bfef-ad67-40a2-a313-0b154a63945d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:44:08.244332 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c8d5406-fa16-42ae-9452-58cea9c7b4ec None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:44:08.244974 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c8d5406-fa16-42ae-9452-58cea9c7b4ec None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:44:08.245313 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c8d5406-fa16-42ae-9452-58cea9c7b4ec None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:44:08.245625 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c8d5406-fa16-42ae-9452-58cea9c7b4ec None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:44:08.255979 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-8be3bfef-ad67-40a2-a313-0b154a63945d None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 01:44:08.257368 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c8d5406-fa16-42ae-9452-58cea9c7b4ec None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:44:08.257852 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-8be3bfef-ad67-40a2-a313-0b154a63945d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:44:08.258844 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c8d5406-fa16-42ae-9452-58cea9c7b4ec None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:44:08.277991 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application [None req-0c8d5406-fa16-42ae-9452-58cea9c7b4ec None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 01:44:08.277991 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 01:44:08.277991 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 01:44:08.277991 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 01:44:08.277991 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 01:44:08.277991 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 01:44:08.277991 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 01:44:08.277991 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 01:44:08.277991 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 01:44:08.277991 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 01:44:08.277991 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 01:44:08.277991 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 01:44:08.277991 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 30 01:44:08.277991 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 30 01:44:08.277991 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 30 01:44:08.277991 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 30 01:44:08.278979 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 01:44:08.278979 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 01:44:08.278979 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 01:44:08.278979 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 01:44:08.278979 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 01:44:08.278979 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 01:44:08.278979 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 01:44:08.278979 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application with Lock( Aug 30 01:44:08.278979 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 01:44:08.278979 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._enter() Aug 30 01:44:08.278979 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 01:44:08.278979 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 01:44:08.278979 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 01:44:08.278979 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.creator() Aug 30 01:44:08.278979 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 01:44:08.278979 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application created_value = creator( Aug 30 01:44:08.278979 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 30 01:44:08.280225 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 30 01:44:08.280225 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 01:44:08.280225 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application Aug 30 01:44:08.280225 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 482/971] 10.222.0.27 () {56 vars in 1108 bytes} [Tue Aug 30 01:44:08 2022] GET /identity/v3/domains/Default => generated 87 bytes in 187 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 01:44:08.284895 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c8d5406-fa16-42ae-9452-58cea9c7b4ec None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:44:08.298851 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:44:08.326123 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:44:08.349761 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 490/972] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Aug 30 01:44:08 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Aug 30 01:44:08.399192 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c8d5406-fa16-42ae-9452-58cea9c7b4ec None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:44:08.436401 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60f16eb1-309f-4097-b9f4-12748eef284b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:44:08.437011 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60f16eb1-309f-4097-b9f4-12748eef284b None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:44:08.437356 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60f16eb1-309f-4097-b9f4-12748eef284b None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:44:08.437661 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60f16eb1-309f-4097-b9f4-12748eef284b None admin] PATH_INFO: `/v3/domains` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:44:08.438313 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60f16eb1-309f-4097-b9f4-12748eef284b None admin] RBAC: Adding query filter params (name=Default) {{(pid=81578) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:44:08.438620 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60f16eb1-309f-4097-b9f4-12748eef284b None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:44:08.439981 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60f16eb1-309f-4097-b9f4-12748eef284b None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:44:08.454480 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 483/973] 10.222.0.27 () {56 vars in 1109 bytes} [Tue Aug 30 01:44:08 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 30 01:44:08.460844 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8be3bfef-ad67-40a2-a313-0b154a63945d None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:44:08.572472 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8be3bfef-ad67-40a2-a313-0b154a63945d None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:44:08.613872 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96b07efb-a0ec-4095-aed0-a2d80a55c822 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:44:08.614426 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96b07efb-a0ec-4095-aed0-a2d80a55c822 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:44:08.614707 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96b07efb-a0ec-4095-aed0-a2d80a55c822 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:44:08.614978 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96b07efb-a0ec-4095-aed0-a2d80a55c822 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:44:08.622042 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96b07efb-a0ec-4095-aed0-a2d80a55c822 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:44:08.623496 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96b07efb-a0ec-4095-aed0-a2d80a55c822 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:44:08.634795 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application [None req-96b07efb-a0ec-4095-aed0-a2d80a55c822 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 30 01:44:08.634795 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 01:44:08.634795 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 01:44:08.634795 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 01:44:08.634795 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 01:44:08.634795 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 01:44:08.634795 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 01:44:08.634795 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 01:44:08.634795 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 01:44:08.634795 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 01:44:08.634795 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 01:44:08.634795 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 01:44:08.634795 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 30 01:44:08.634795 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 30 01:44:08.634795 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 30 01:44:08.636878 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 30 01:44:08.636878 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 01:44:08.636878 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 01:44:08.636878 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 01:44:08.636878 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 01:44:08.636878 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 01:44:08.636878 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 01:44:08.636878 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 01:44:08.636878 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application with Lock( Aug 30 01:44:08.636878 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 01:44:08.636878 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._enter() Aug 30 01:44:08.636878 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 01:44:08.636878 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 01:44:08.636878 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 01:44:08.636878 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.creator() Aug 30 01:44:08.636878 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 01:44:08.636878 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application created_value = creator( Aug 30 01:44:08.638148 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 30 01:44:08.638148 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 30 01:44:08.638148 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 30 01:44:08.638148 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 30 01:44:08.638148 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 30 01:44:08.638148 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 30 01:44:08.638148 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 30 01:44:08.638148 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Aug 30 01:44:08.638148 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 491/974] 10.222.0.27 () {56 vars in 1111 bytes} [Tue Aug 30 01:44:08 2022] GET /identity/v3/projects/service => generated 88 bytes in 176 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 01:44:08.641721 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60f16eb1-309f-4097-b9f4-12748eef284b None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:44:08.648342 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96b07efb-a0ec-4095-aed0-a2d80a55c822 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:44:08.710618 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:44:08.719336 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:44:08.758166 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60f16eb1-309f-4097-b9f4-12748eef284b None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:44:08.780563 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96b07efb-a0ec-4095-aed0-a2d80a55c822 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:44:08.800221 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05c5c56b-7d06-41b8-845b-dbae6f041dfc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:44:08.801122 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05c5c56b-7d06-41b8-845b-dbae6f041dfc None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:44:08.801834 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05c5c56b-7d06-41b8-845b-dbae6f041dfc None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:44:08.802262 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05c5c56b-7d06-41b8-845b-dbae6f041dfc None admin] PATH_INFO: `/v3/projects` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:44:08.803183 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05c5c56b-7d06-41b8-845b-dbae6f041dfc None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81578) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:44:08.803639 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05c5c56b-7d06-41b8-845b-dbae6f041dfc None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:44:08.805213 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05c5c56b-7d06-41b8-845b-dbae6f041dfc None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:44:08.822985 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 484/975] 10.222.0.27 () {56 vars in 1148 bytes} [Tue Aug 30 01:44:08 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:44:08.824498 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05c5c56b-7d06-41b8-845b-dbae6f041dfc None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:44:08.827862 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a66f3cd9-6c53-4655-8dce-a3321c6afcf5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:44:08.828480 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a66f3cd9-6c53-4655-8dce-a3321c6afcf5 None admin] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:44:08.828759 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a66f3cd9-6c53-4655-8dce-a3321c6afcf5 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:44:08.829028 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a66f3cd9-6c53-4655-8dce-a3321c6afcf5 None admin] PATH_INFO: `/v3/users` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:44:08.830601 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a66f3cd9-6c53-4655-8dce-a3321c6afcf5 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:44:08.832045 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a66f3cd9-6c53-4655-8dce-a3321c6afcf5 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:44:08.875007 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.sql.core [None req-a66f3cd9-6c53-4655-8dce-a3321c6afcf5 None admin] Conflict user: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'default-cinder' for key 'local_user.domain_id'") Aug 30 01:44:08.875007 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [SQL: INSERT INTO local_user (user_id, domain_id, name, failed_auth_count, failed_auth_at) VALUES (%(user_id)s, %(domain_id)s, %(name)s, %(failed_auth_count)s, %(failed_auth_at)s)] Aug 30 01:44:08.875007 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [parameters: {'user_id': '4dd4df1351fa43e391d895492414ca20', 'domain_id': 'default', 'name': 'cinder', 'failed_auth_count': 0, 'failed_auth_at': None}] Aug 30 01:44:08.875007 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: (Background on this error at: https://sqlalche.me/e/14/gkpj) {{(pid=81577) wrapper /opt/stack/keystone/keystone/common/sql/core.py:533}} Aug 30 01:44:08.882563 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application [None req-a66f3cd9-6c53-4655-8dce-a3321c6afcf5 None admin] Conflict occurred attempting to store user - Duplicate entry found with name cinder at domain ID default.: keystone.exception.Conflict: Conflict occurred attempting to store user - Duplicate entry found with name cinder at domain ID default. Aug 30 01:44:08.882563 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 01:44:08.882563 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Aug 30 01:44:08.882563 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application self.dialect.do_execute( Aug 30 01:44:08.882563 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Aug 30 01:44:08.882563 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Aug 30 01:44:08.882563 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Aug 30 01:44:08.882563 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application result = self._query(query) Aug 30 01:44:08.882563 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Aug 30 01:44:08.882563 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application conn.query(q) Aug 30 01:44:08.882563 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Aug 30 01:44:08.882563 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Aug 30 01:44:08.882563 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Aug 30 01:44:08.882563 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application result.read() Aug 30 01:44:08.882563 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Aug 30 01:44:08.883536 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Aug 30 01:44:08.883536 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Aug 30 01:44:08.883536 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application packet.raise_for_error() Aug 30 01:44:08.883536 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Aug 30 01:44:08.883536 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Aug 30 01:44:08.883536 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Aug 30 01:44:08.883536 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Aug 30 01:44:08.883536 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application pymysql.err.IntegrityError: (1062, "Duplicate entry 'default-cinder' for key 'local_user.domain_id'") Aug 30 01:44:08.883536 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Aug 30 01:44:08.883536 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application The above exception was the direct cause of the following exception: Aug 30 01:44:08.883536 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Aug 30 01:44:08.883536 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 01:44:08.883536 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 528, in wrapper Aug 30 01:44:08.883536 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return method(*args, **kwargs) Aug 30 01:44:08.883536 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 141, in create_user Aug 30 01:44:08.883536 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return base.filter_user(user_ref.to_dict()) Aug 30 01:44:08.883536 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Aug 30 01:44:08.883536 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application next(self.gen) Aug 30 01:44:08.884538 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 1060, in _transaction_scope Aug 30 01:44:08.884538 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application yield resource Aug 30 01:44:08.884538 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Aug 30 01:44:08.884538 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application next(self.gen) Aug 30 01:44:08.884538 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 662, in _session Aug 30 01:44:08.884538 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application self.session.rollback() Aug 30 01:44:08.884538 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Aug 30 01:44:08.884538 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application self.force_reraise() Aug 30 01:44:08.884538 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Aug 30 01:44:08.884538 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise self.value Aug 30 01:44:08.884538 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 659, in _session Aug 30 01:44:08.884538 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application self._end_session_transaction(self.session) Aug 30 01:44:08.884538 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 687, in _end_session_transaction Aug 30 01:44:08.884538 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application session.commit() Aug 30 01:44:08.884538 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 1451, in commit Aug 30 01:44:08.884538 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application self._transaction.commit(_to_root=self.future) Aug 30 01:44:08.884538 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 829, in commit Aug 30 01:44:08.885508 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application self._prepare_impl() Aug 30 01:44:08.885508 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 808, in _prepare_impl Aug 30 01:44:08.885508 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application self.session.flush() Aug 30 01:44:08.885508 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3383, in flush Aug 30 01:44:08.885508 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application self._flush(objects) Aug 30 01:44:08.885508 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3523, in _flush Aug 30 01:44:08.885508 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application transaction.rollback(_capture_exception=True) Aug 30 01:44:08.885508 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__ Aug 30 01:44:08.885508 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application compat.raise_( Aug 30 01:44:08.885508 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Aug 30 01:44:08.885508 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise exception Aug 30 01:44:08.885508 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3483, in _flush Aug 30 01:44:08.885508 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application flush_context.execute() Aug 30 01:44:08.885508 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 456, in execute Aug 30 01:44:08.885508 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application rec.execute(self) Aug 30 01:44:08.885508 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 630, in execute Aug 30 01:44:08.885508 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application util.preloaded.orm_persistence.save_obj( Aug 30 01:44:08.886455 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 245, in save_obj Aug 30 01:44:08.886455 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application _emit_insert_statements( Aug 30 01:44:08.886455 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 1238, in _emit_insert_statements Aug 30 01:44:08.886455 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application result = connection._execute_20( Aug 30 01:44:08.886455 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1705, in _execute_20 Aug 30 01:44:08.886455 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return meth(self, args_10style, kwargs_10style, execution_options) Aug 30 01:44:08.886455 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/sql/elements.py", line 333, in _execute_on_connection Aug 30 01:44:08.886455 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return connection._execute_clauseelement( Aug 30 01:44:08.886455 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1572, in _execute_clauseelement Aug 30 01:44:08.886455 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application ret = self._execute_context( Aug 30 01:44:08.886455 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1943, in _execute_context Aug 30 01:44:08.886455 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application self._handle_dbapi_exception( Aug 30 01:44:08.886455 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 2122, in _handle_dbapi_exception Aug 30 01:44:08.886455 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application util.raise_(newraise, with_traceback=exc_info[2], from_=e) Aug 30 01:44:08.886455 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Aug 30 01:44:08.886455 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise exception Aug 30 01:44:08.887331 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Aug 30 01:44:08.887331 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application self.dialect.do_execute( Aug 30 01:44:08.887331 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Aug 30 01:44:08.887331 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Aug 30 01:44:08.887331 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Aug 30 01:44:08.887331 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application result = self._query(query) Aug 30 01:44:08.887331 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Aug 30 01:44:08.887331 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application conn.query(q) Aug 30 01:44:08.887331 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Aug 30 01:44:08.887331 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Aug 30 01:44:08.887331 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Aug 30 01:44:08.887331 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application result.read() Aug 30 01:44:08.887331 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Aug 30 01:44:08.887331 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Aug 30 01:44:08.887331 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Aug 30 01:44:08.887331 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application packet.raise_for_error() Aug 30 01:44:08.888184 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Aug 30 01:44:08.888184 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Aug 30 01:44:08.888184 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Aug 30 01:44:08.888184 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Aug 30 01:44:08.888184 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application oslo_db.exception.DBDuplicateEntry: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'default-cinder' for key 'local_user.domain_id'") Aug 30 01:44:08.888184 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application [SQL: INSERT INTO local_user (user_id, domain_id, name, failed_auth_count, failed_auth_at) VALUES (%(user_id)s, %(domain_id)s, %(name)s, %(failed_auth_count)s, %(failed_auth_at)s)] Aug 30 01:44:08.888184 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application [parameters: {'user_id': '4dd4df1351fa43e391d895492414ca20', 'domain_id': 'default', 'name': 'cinder', 'failed_auth_count': 0, 'failed_auth_at': None}] Aug 30 01:44:08.888184 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application (Background on this error at: https://sqlalche.me/e/14/gkpj) Aug 30 01:44:08.888184 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Aug 30 01:44:08.888184 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 30 01:44:08.888184 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Aug 30 01:44:08.888184 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 01:44:08.888184 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 01:44:08.888184 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 01:44:08.888184 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 01:44:08.889003 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 01:44:08.889003 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 01:44:08.889003 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 01:44:08.889003 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 01:44:08.889003 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 01:44:08.889003 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 01:44:08.889003 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 01:44:08.889003 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 244, in post Aug 30 01:44:08.889003 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.create_user( Aug 30 01:44:08.889003 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 01:44:08.889003 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 01:44:08.889003 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 30 01:44:08.889003 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 30 01:44:08.889003 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 30 01:44:08.889003 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 30 01:44:08.889003 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1000, in create_user Aug 30 01:44:08.889003 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application ref = self._create_user_with_federated_objects(user, driver) Aug 30 01:44:08.891946 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 966, in _create_user_with_federated_objects Aug 30 01:44:08.891946 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application user = driver.create_user(user['id'], user) Aug 30 01:44:08.891946 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 563, in wrapper Aug 30 01:44:08.891946 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise exception.Conflict(type=conflict_type, Aug 30 01:44:08.891946 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application keystone.exception.Conflict: Conflict occurred attempting to store user - Duplicate entry found with name cinder at domain ID default. Aug 30 01:44:08.891946 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Aug 30 01:44:08.891946 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 492/976] 10.222.0.27 () {60 vars in 1131 bytes} [Tue Aug 30 01:44:08 2022] POST /identity/v3/users => generated 160 bytes in 237 msecs (HTTP/1.1 409) 5 headers in 183 bytes (2 switches on core 0) Aug 30 01:44:08.891946 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a66f3cd9-6c53-4655-8dce-a3321c6afcf5 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:44:08.967574 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05c5c56b-7d06-41b8-845b-dbae6f041dfc None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:44:09.003346 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a66f3cd9-6c53-4655-8dce-a3321c6afcf5 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:44:09.007764 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:44:09.014891 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19eea0e0-ae47-470e-abf6-8d96b2bd859a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:44:09.019434 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19eea0e0-ae47-470e-abf6-8d96b2bd859a None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:44:09.019434 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19eea0e0-ae47-470e-abf6-8d96b2bd859a None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:44:09.019636 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:44:09.019752 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19eea0e0-ae47-470e-abf6-8d96b2bd859a None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:44:09.019752 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19eea0e0-ae47-470e-abf6-8d96b2bd859a None admin] RBAC: Adding query filter params (role.id=8e1192374afc4af6a7cea01c69e72402, scope.project.id=4562f3951fbe465280073e60a11e44e2, user.id=121c374d49a64335ab74a431cfbc1320) {{(pid=81578) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:44:09.019752 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19eea0e0-ae47-470e-abf6-8d96b2bd859a None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:44:09.020700 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19eea0e0-ae47-470e-abf6-8d96b2bd859a None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:44:09.033881 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 485/977] 10.222.0.27 () {56 vars in 1374 bytes} [Tue Aug 30 01:44:08 2022] GET /identity/v3/role_assignments?user.id=121c374d49a64335ab74a431cfbc1320&scope.project.id=4562f3951fbe465280073e60a11e44e2&role.id=8e1192374afc4af6a7cea01c69e72402 => generated 608 bytes in 205 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:44:09.050209 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-208eaa77-a0e3-43ef-ba7c-b9aed6996bce None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:44:09.050859 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-208eaa77-a0e3-43ef-ba7c-b9aed6996bce None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:44:09.051708 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-208eaa77-a0e3-43ef-ba7c-b9aed6996bce None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:44:09.051981 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-208eaa77-a0e3-43ef-ba7c-b9aed6996bce None admin] PATH_INFO: `/v3/users/cinder` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:44:09.060546 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-208eaa77-a0e3-43ef-ba7c-b9aed6996bce None admin] RBAC: Authorizing `identity:get_user(user_id=cinder)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:44:09.064011 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-208eaa77-a0e3-43ef-ba7c-b9aed6996bce None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:44:09.077617 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application [None req-208eaa77-a0e3-43ef-ba7c-b9aed6996bce None admin] Could not find user: cinder.: keystone.exception.UserNotFound: Could not find user: cinder. Aug 30 01:44:09.077617 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 01:44:09.077617 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 01:44:09.077617 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 01:44:09.077617 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 01:44:09.077617 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 01:44:09.077617 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 01:44:09.077617 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 01:44:09.077617 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 01:44:09.077617 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 01:44:09.077617 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 01:44:09.077617 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 01:44:09.077617 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 30 01:44:09.077617 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 30 01:44:09.077617 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 30 01:44:09.077617 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 30 01:44:09.078676 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 01:44:09.078676 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 01:44:09.078676 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 30 01:44:09.078676 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 30 01:44:09.078676 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 30 01:44:09.078676 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 30 01:44:09.078676 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 01:44:09.078676 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 01:44:09.078676 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 01:44:09.078676 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 01:44:09.078676 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 01:44:09.078676 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application with Lock( Aug 30 01:44:09.078676 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 01:44:09.078676 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._enter() Aug 30 01:44:09.078676 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 01:44:09.078676 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 01:44:09.078676 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 01:44:09.078676 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.creator() Aug 30 01:44:09.080214 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 01:44:09.080214 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application created_value = creator( Aug 30 01:44:09.080214 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 30 01:44:09.080214 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 30 01:44:09.080214 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 30 01:44:09.080214 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 30 01:44:09.080214 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 30 01:44:09.080214 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 30 01:44:09.080214 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: cinder. Aug 30 01:44:09.080214 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Aug 30 01:44:09.080214 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 493/978] 10.222.0.27 () {56 vars in 1099 bytes} [Tue Aug 30 01:44:08 2022] GET /identity/v3/users/cinder => generated 84 bytes in 189 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 01:44:09.085346 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19eea0e0-ae47-470e-abf6-8d96b2bd859a None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:44:09.205788 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19eea0e0-ae47-470e-abf6-8d96b2bd859a None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:44:09.245911 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6449b767-d4f2-4ced-a2d9-5b3d8c34b251 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:44:09.246576 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6449b767-d4f2-4ced-a2d9-5b3d8c34b251 None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:44:09.246864 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6449b767-d4f2-4ced-a2d9-5b3d8c34b251 None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:44:09.247244 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6449b767-d4f2-4ced-a2d9-5b3d8c34b251 None admin] PATH_INFO: `/v3/users` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:44:09.247951 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6449b767-d4f2-4ced-a2d9-5b3d8c34b251 None admin] RBAC: Adding query filter params (name=cinder) {{(pid=81578) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:44:09.248253 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6449b767-d4f2-4ced-a2d9-5b3d8c34b251 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:44:09.249697 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6449b767-d4f2-4ced-a2d9-5b3d8c34b251 None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:44:09.287939 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 486/979] 10.222.0.27 () {56 vars in 1101 bytes} [Tue Aug 30 01:44:09 2022] GET /identity/v3/users?name=cinder => generated 354 bytes in 203 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:44:09.442518 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8f1c64f-24a2-4d87-8a67-55e2c51efe5d None None] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:44:09.443034 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8f1c64f-24a2-4d87-8a67-55e2c51efe5d None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:44:09.443034 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8f1c64f-24a2-4d87-8a67-55e2c51efe5d None None] PATH_INFO: `/` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:44:09.447216 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 494/980] 10.222.0.27 () {54 vars in 927 bytes} [Tue Aug 30 01:44:09 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 01:44:09.452716 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16f44c9f-35cf-4968-92d4-ade6c69c4ed0 None None] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:44:09.452891 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5478b23b-b025-4c03-b090-519f63607197 None None] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:44:09.452974 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16f44c9f-35cf-4968-92d4-ade6c69c4ed0 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:44:09.453142 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5478b23b-b025-4c03-b090-519f63607197 None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:44:09.453230 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16f44c9f-35cf-4968-92d4-ade6c69c4ed0 None None] PATH_INFO: `/` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:44:09.453745 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5478b23b-b025-4c03-b090-519f63607197 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:44:09.454862 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 495/981] 10.222.0.27 () {54 vars in 927 bytes} [Tue Aug 30 01:44:09 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 01:44:09.461306 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b40fccf1-31dd-4e40-82d3-daa63419eb28 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:44:09.461306 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b40fccf1-31dd-4e40-82d3-daa63419eb28 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:44:09.461306 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b40fccf1-31dd-4e40-82d3-daa63419eb28 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:44:09.479022 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.auth.core [None req-5478b23b-b025-4c03-b090-519f63607197 None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81578) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 01:44:09.480222 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-5478b23b-b025-4c03-b090-519f63607197 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:44:09.480401 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-b40fccf1-31dd-4e40-82d3-daa63419eb28 None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 01:44:09.481918 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-b40fccf1-31dd-4e40-82d3-daa63419eb28 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:44:09.543664 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:44:09.570813 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 496/982] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Aug 30 01:44:09 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 111 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Aug 30 01:44:09.578799 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 487/983] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Aug 30 01:44:09 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 127 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 30 01:44:09.840409 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b40fccf1-31dd-4e40-82d3-daa63419eb28 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:44:09.849152 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5478b23b-b025-4c03-b090-519f63607197 None None] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:44:09.943516 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:44:09.946647 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b40fccf1-31dd-4e40-82d3-daa63419eb28 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:44:09.953239 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5478b23b-b025-4c03-b090-519f63607197 None None] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:44:09.985303 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-857c4274-39a1-4fc0-8de4-e5df094d0f40 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:44:09.986199 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-857c4274-39a1-4fc0-8de4-e5df094d0f40 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:44:09.986626 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-857c4274-39a1-4fc0-8de4-e5df094d0f40 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:44:09.987032 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-857c4274-39a1-4fc0-8de4-e5df094d0f40 None admin] PATH_INFO: `/v3/services/object-store` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:44:09.987859 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-857c4274-39a1-4fc0-8de4-e5df094d0f40 None admin] RBAC: Authorizing `identity:get_service(service_id=object-store)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:44:09.989677 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-857c4274-39a1-4fc0-8de4-e5df094d0f40 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:44:09.991128 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-589a2de1-9254-4bae-9bf3-3879ebf97099 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:44:09.991735 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-589a2de1-9254-4bae-9bf3-3879ebf97099 None admin] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:44:09.992018 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-589a2de1-9254-4bae-9bf3-3879ebf97099 None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:44:09.992290 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-589a2de1-9254-4bae-9bf3-3879ebf97099 None admin] PATH_INFO: `/v3/services` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:44:09.992941 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-589a2de1-9254-4bae-9bf3-3879ebf97099 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:44:09.994345 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-589a2de1-9254-4bae-9bf3-3879ebf97099 None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:44:09.999866 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application [None req-857c4274-39a1-4fc0-8de4-e5df094d0f40 None admin] Could not find service: object-store.: keystone.exception.ServiceNotFound: Could not find service: object-store. Aug 30 01:44:09.999866 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 01:44:09.999866 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Aug 30 01:44:09.999866 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Aug 30 01:44:09.999866 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Aug 30 01:44:09.999866 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Aug 30 01:44:09.999866 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Aug 30 01:44:09.999866 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 30 01:44:09.999866 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Aug 30 01:44:09.999866 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Aug 30 01:44:09.999866 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 30 01:44:09.999866 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Aug 30 01:44:09.999866 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 01:44:09.999866 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 01:44:09.999866 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 01:44:09.999866 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 01:44:09.999866 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 01:44:10.001113 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 01:44:10.001113 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 01:44:10.001113 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 01:44:10.001113 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 01:44:10.001113 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 01:44:10.001113 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 01:44:10.001113 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Aug 30 01:44:10.001113 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_service(service_id) Aug 30 01:44:10.001113 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Aug 30 01:44:10.001113 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Aug 30 01:44:10.001113 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 01:44:10.001113 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 01:44:10.001113 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 01:44:10.001113 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 01:44:10.001113 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 01:44:10.001113 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 01:44:10.002191 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 01:44:10.002191 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application with Lock( Aug 30 01:44:10.002191 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 01:44:10.002191 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._enter() Aug 30 01:44:10.002191 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 01:44:10.002191 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 01:44:10.002191 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 01:44:10.002191 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.creator() Aug 30 01:44:10.002191 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 01:44:10.002191 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application created_value = creator( Aug 30 01:44:10.002191 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Aug 30 01:44:10.002191 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 30 01:44:10.002191 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Aug 30 01:44:10.002191 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Aug 30 01:44:10.005609 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 497/984] 10.222.0.27 () {56 vars in 1126 bytes} [Tue Aug 30 01:44:09 2022] GET /identity/v3/services/object-store => generated 93 bytes in 166 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 01:44:10.007039 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-857c4274-39a1-4fc0-8de4-e5df094d0f40 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:44:10.059734 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 488/985] 10.222.0.27 () {60 vars in 1141 bytes} [Tue Aug 30 01:44:09 2022] POST /identity/v3/services => generated 242 bytes in 211 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 30 01:44:10.126743 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-857c4274-39a1-4fc0-8de4-e5df094d0f40 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:44:10.171761 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be33e3a7-687b-4114-a20b-cddc8552137c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:44:10.172382 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be33e3a7-687b-4114-a20b-cddc8552137c None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:44:10.172656 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be33e3a7-687b-4114-a20b-cddc8552137c None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:44:10.172924 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be33e3a7-687b-4114-a20b-cddc8552137c None admin] PATH_INFO: `/v3/services` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:44:10.173592 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be33e3a7-687b-4114-a20b-cddc8552137c None admin] RBAC: Adding query filter params (name=object-store) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:44:10.173891 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be33e3a7-687b-4114-a20b-cddc8552137c None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:44:10.175264 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be33e3a7-687b-4114-a20b-cddc8552137c None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:44:10.186355 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 498/986] 10.222.0.27 () {56 vars in 1122 bytes} [Tue Aug 30 01:44:10 2022] GET /identity/v3/services?name=object-store => generated 129 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 30 01:44:10.187414 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-589a2de1-9254-4bae-9bf3-3879ebf97099 None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:44:10.236873 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:44:10.299991 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-589a2de1-9254-4bae-9bf3-3879ebf97099 None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:44:10.334121 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-903817bf-8d63-4ebb-ab7e-8ed2d5bc45cc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:44:10.334728 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-903817bf-8d63-4ebb-ab7e-8ed2d5bc45cc None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:44:10.335007 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-903817bf-8d63-4ebb-ab7e-8ed2d5bc45cc None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:44:10.335272 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-903817bf-8d63-4ebb-ab7e-8ed2d5bc45cc None admin] PATH_INFO: `/v3/services` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:44:10.335900 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-903817bf-8d63-4ebb-ab7e-8ed2d5bc45cc None admin] RBAC: Adding query filter params (type=object-store) {{(pid=81578) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:44:10.336208 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-903817bf-8d63-4ebb-ab7e-8ed2d5bc45cc None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:44:10.337615 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-903817bf-8d63-4ebb-ab7e-8ed2d5bc45cc None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:44:10.342535 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 489/987] 10.222.0.27 () {56 vars in 1122 bytes} [Tue Aug 30 01:44:10 2022] GET /identity/v3/services?type=object-store => generated 356 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:44:10.349000 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be33e3a7-687b-4114-a20b-cddc8552137c None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:44:10.375443 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:44:10.451522 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be33e3a7-687b-4114-a20b-cddc8552137c None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:44:10.485900 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b62cda4-f94b-4397-b865-0beddb6ace83 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:44:10.486507 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b62cda4-f94b-4397-b865-0beddb6ace83 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:44:10.486785 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b62cda4-f94b-4397-b865-0beddb6ace83 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:44:10.487108 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b62cda4-f94b-4397-b865-0beddb6ace83 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:44:10.487762 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b62cda4-f94b-4397-b865-0beddb6ace83 None admin] RBAC: Adding query filter params (interface=admin, service_id=54dcf8b2a2e74b31b6293369850f6fdf, region_id=RegionOne) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:44:10.488071 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b62cda4-f94b-4397-b865-0beddb6ace83 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:44:10.490066 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b62cda4-f94b-4397-b865-0beddb6ace83 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:44:10.499939 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 499/988] 10.222.0.27 () {56 vars in 1249 bytes} [Tue Aug 30 01:44:10 2022] GET /identity/v3/endpoints?service_id=54dcf8b2a2e74b31b6293369850f6fdf&interface=admin®ion_id=RegionOne => generated 193 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:44:10.501186 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-903817bf-8d63-4ebb-ab7e-8ed2d5bc45cc None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:44:10.606676 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-903817bf-8d63-4ebb-ab7e-8ed2d5bc45cc None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:44:10.614268 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:44:10.640802 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ba3201b-87be-4cd8-ac0d-5058f2dcdc00 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:44:10.641404 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ba3201b-87be-4cd8-ac0d-5058f2dcdc00 None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:44:10.641711 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ba3201b-87be-4cd8-ac0d-5058f2dcdc00 None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:44:10.641980 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ba3201b-87be-4cd8-ac0d-5058f2dcdc00 None admin] PATH_INFO: `/v3/services` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:44:10.642585 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ba3201b-87be-4cd8-ac0d-5058f2dcdc00 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:44:10.643928 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ba3201b-87be-4cd8-ac0d-5058f2dcdc00 None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:44:10.650935 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 490/989] 10.222.0.27 () {56 vars in 1087 bytes} [Tue Aug 30 01:44:10 2022] GET /identity/v3/services => generated 1727 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 01:44:11.062996 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8b12d70-dbc8-4ab0-a42d-e9730d71966a None None] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:44:11.063375 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8b12d70-dbc8-4ab0-a42d-e9730d71966a None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:44:11.063605 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8b12d70-dbc8-4ab0-a42d-e9730d71966a None None] PATH_INFO: `/` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:44:11.065025 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 500/990] 10.222.0.27 () {54 vars in 927 bytes} [Tue Aug 30 01:44:11 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 01:44:11.070768 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f89199ba-6e88-452e-bc7d-d91185a06cfe None None] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:44:11.071058 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f89199ba-6e88-452e-bc7d-d91185a06cfe None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:44:11.071331 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f89199ba-6e88-452e-bc7d-d91185a06cfe None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:44:11.089416 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.auth.core [None req-f89199ba-6e88-452e-bc7d-d91185a06cfe None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81578) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 01:44:11.090622 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-f89199ba-6e88-452e-bc7d-d91185a06cfe None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:44:11.159478 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 491/991] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Aug 30 01:44:11 2022] POST /identity/v3/auth/tokens => generated 1570 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 30 01:44:11.427945 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8b12d70-dbc8-4ab0-a42d-e9730d71966a None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:44:11.517153 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:44:11.531682 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8b12d70-dbc8-4ab0-a42d-e9730d71966a None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:44:11.567501 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9508b69-e0ac-4d1d-8005-0c06733a128c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:44:11.568123 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9508b69-e0ac-4d1d-8005-0c06733a128c None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:44:11.568397 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9508b69-e0ac-4d1d-8005-0c06733a128c None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:44:11.568660 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9508b69-e0ac-4d1d-8005-0c06733a128c None admin] PATH_INFO: `/v3/services/image` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:44:11.569235 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9508b69-e0ac-4d1d-8005-0c06733a128c None admin] RBAC: Authorizing `identity:get_service(service_id=image)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:44:11.570637 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9508b69-e0ac-4d1d-8005-0c06733a128c None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:44:11.583882 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application [None req-f9508b69-e0ac-4d1d-8005-0c06733a128c None admin] Could not find service: image.: keystone.exception.ServiceNotFound: Could not find service: image. Aug 30 01:44:11.583882 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 01:44:11.583882 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Aug 30 01:44:11.583882 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Aug 30 01:44:11.583882 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Aug 30 01:44:11.583882 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Aug 30 01:44:11.583882 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Aug 30 01:44:11.583882 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 30 01:44:11.583882 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Aug 30 01:44:11.583882 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Aug 30 01:44:11.583882 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 30 01:44:11.583882 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Aug 30 01:44:11.583882 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 01:44:11.583882 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 01:44:11.583882 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 01:44:11.583882 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 01:44:11.583882 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 01:44:11.584947 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 01:44:11.584947 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 01:44:11.584947 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 01:44:11.584947 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 01:44:11.584947 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 01:44:11.584947 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 01:44:11.584947 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Aug 30 01:44:11.584947 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_service(service_id) Aug 30 01:44:11.584947 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Aug 30 01:44:11.584947 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Aug 30 01:44:11.584947 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 01:44:11.584947 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 01:44:11.584947 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 01:44:11.584947 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 01:44:11.584947 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 01:44:11.584947 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 01:44:11.590363 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 01:44:11.590363 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application with Lock( Aug 30 01:44:11.590363 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 01:44:11.590363 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._enter() Aug 30 01:44:11.590363 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 01:44:11.590363 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 01:44:11.590363 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 01:44:11.590363 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.creator() Aug 30 01:44:11.590363 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 01:44:11.590363 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application created_value = creator( Aug 30 01:44:11.590363 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Aug 30 01:44:11.590363 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 30 01:44:11.590363 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Aug 30 01:44:11.590363 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Aug 30 01:44:11.590363 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 501/992] 10.222.0.27 () {56 vars in 1105 bytes} [Tue Aug 30 01:44:11 2022] GET /identity/v3/services/image => generated 86 bytes in 158 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 01:44:11.592392 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f89199ba-6e88-452e-bc7d-d91185a06cfe None None] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:44:11.691890 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:44:11.695437 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f89199ba-6e88-452e-bc7d-d91185a06cfe None None] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:44:11.740389 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86d0019c-8ae6-4a4d-9b89-74848c52f35e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:44:11.741078 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86d0019c-8ae6-4a4d-9b89-74848c52f35e None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:44:11.741354 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86d0019c-8ae6-4a4d-9b89-74848c52f35e None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:44:11.741691 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86d0019c-8ae6-4a4d-9b89-74848c52f35e None admin] PATH_INFO: `/v3/services` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:44:11.742336 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86d0019c-8ae6-4a4d-9b89-74848c52f35e None admin] RBAC: Adding query filter params (name=image) {{(pid=81578) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:44:11.742640 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86d0019c-8ae6-4a4d-9b89-74848c52f35e None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:44:11.744006 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86d0019c-8ae6-4a4d-9b89-74848c52f35e None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:44:11.749825 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 492/993] 10.222.0.27 () {56 vars in 1108 bytes} [Tue Aug 30 01:44:11 2022] GET /identity/v3/services?name=image => generated 122 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:44:11.755824 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9508b69-e0ac-4d1d-8005-0c06733a128c None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:44:11.858060 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9508b69-e0ac-4d1d-8005-0c06733a128c None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:44:11.892071 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aacb7b96-ca86-4679-9c11-9c8f972ec0ec None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:44:11.892688 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aacb7b96-ca86-4679-9c11-9c8f972ec0ec None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:44:11.892969 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aacb7b96-ca86-4679-9c11-9c8f972ec0ec None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:44:11.893242 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aacb7b96-ca86-4679-9c11-9c8f972ec0ec None admin] PATH_INFO: `/v3/services` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:44:11.893897 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aacb7b96-ca86-4679-9c11-9c8f972ec0ec None admin] RBAC: Adding query filter params (type=image) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:44:11.894201 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aacb7b96-ca86-4679-9c11-9c8f972ec0ec None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:44:11.895717 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aacb7b96-ca86-4679-9c11-9c8f972ec0ec None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:44:11.900350 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 502/994] 10.222.0.27 () {56 vars in 1108 bytes} [Tue Aug 30 01:44:11 2022] GET /identity/v3/services?type=image => generated 122 bytes in 145 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:44:12.012974 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcf62274-2c18-4706-bf71-7e7d97adfbb9 None None] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:44:12.013177 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcf62274-2c18-4706-bf71-7e7d97adfbb9 None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:44:12.013440 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcf62274-2c18-4706-bf71-7e7d97adfbb9 None None] PATH_INFO: `/` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:44:12.015049 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 493/995] 10.222.0.27 () {54 vars in 927 bytes} [Tue Aug 30 01:44:12 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 01:44:12.021248 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d897c53-8758-41aa-b980-346fc010eb90 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:44:12.022010 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d897c53-8758-41aa-b980-346fc010eb90 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:44:12.022010 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d897c53-8758-41aa-b980-346fc010eb90 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:44:12.042077 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-2d897c53-8758-41aa-b980-346fc010eb90 None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 01:44:12.043278 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-2d897c53-8758-41aa-b980-346fc010eb90 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:44:12.055763 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:44:12.113626 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 503/996] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Aug 30 01:44:12 2022] POST /identity/v3/auth/tokens => generated 1570 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 30 01:44:12.385427 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcf62274-2c18-4706-bf71-7e7d97adfbb9 None None] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:44:12.488839 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcf62274-2c18-4706-bf71-7e7d97adfbb9 None None] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:44:12.522822 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32fdd55e-81f4-4c67-8c68-ea74c890a408 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:44:12.523452 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32fdd55e-81f4-4c67-8c68-ea74c890a408 None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:44:12.523724 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32fdd55e-81f4-4c67-8c68-ea74c890a408 None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:44:12.523991 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32fdd55e-81f4-4c67-8c68-ea74c890a408 None admin] PATH_INFO: `/v3/services/compute` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:44:12.524565 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32fdd55e-81f4-4c67-8c68-ea74c890a408 None admin] RBAC: Authorizing `identity:get_service(service_id=compute)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:44:12.525941 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32fdd55e-81f4-4c67-8c68-ea74c890a408 None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:44:12.534683 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application [None req-32fdd55e-81f4-4c67-8c68-ea74c890a408 None admin] Could not find service: compute.: keystone.exception.ServiceNotFound: Could not find service: compute. Aug 30 01:44:12.534683 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 01:44:12.534683 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Aug 30 01:44:12.534683 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Aug 30 01:44:12.534683 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Aug 30 01:44:12.534683 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Aug 30 01:44:12.534683 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Aug 30 01:44:12.534683 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 30 01:44:12.534683 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Aug 30 01:44:12.534683 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application Aug 30 01:44:12.534683 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 30 01:44:12.534683 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application Aug 30 01:44:12.534683 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 01:44:12.534683 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 01:44:12.534683 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 01:44:12.534683 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 01:44:12.534683 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 01:44:12.536482 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 01:44:12.536482 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 01:44:12.536482 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 01:44:12.536482 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 01:44:12.536482 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 01:44:12.536482 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 01:44:12.536482 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Aug 30 01:44:12.536482 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._get_service(service_id) Aug 30 01:44:12.536482 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Aug 30 01:44:12.536482 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Aug 30 01:44:12.536482 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 01:44:12.536482 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 01:44:12.536482 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 01:44:12.536482 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 01:44:12.536482 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 01:44:12.536482 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 01:44:12.537359 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 01:44:12.537359 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application with Lock( Aug 30 01:44:12.537359 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 01:44:12.537359 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._enter() Aug 30 01:44:12.537359 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 01:44:12.537359 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 01:44:12.537359 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 01:44:12.537359 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.creator() Aug 30 01:44:12.537359 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 01:44:12.537359 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application created_value = creator( Aug 30 01:44:12.537359 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Aug 30 01:44:12.537359 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 30 01:44:12.537359 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Aug 30 01:44:12.537359 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application Aug 30 01:44:12.537359 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 494/997] 10.222.0.27 () {56 vars in 1111 bytes} [Tue Aug 30 01:44:12 2022] GET /identity/v3/services/compute => generated 88 bytes in 152 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 01:44:12.543336 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d897c53-8758-41aa-b980-346fc010eb90 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:44:12.592734 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0854589f-39bd-4cb3-a5b0-86763cd3683f None None] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:44:12.593912 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0854589f-39bd-4cb3-a5b0-86763cd3683f None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:44:12.594362 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0854589f-39bd-4cb3-a5b0-86763cd3683f None None] PATH_INFO: `/` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:44:12.596767 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 495/998] 10.222.0.27 () {54 vars in 927 bytes} [Tue Aug 30 01:44:12 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 01:44:12.602224 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7747be3-2f28-4952-8bed-9b2c74cd4ec4 None None] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:44:12.602541 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7747be3-2f28-4952-8bed-9b2c74cd4ec4 None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:44:12.602783 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7747be3-2f28-4952-8bed-9b2c74cd4ec4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:44:12.620929 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.auth.core [None req-e7747be3-2f28-4952-8bed-9b2c74cd4ec4 None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81578) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 01:44:12.622143 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-e7747be3-2f28-4952-8bed-9b2c74cd4ec4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:44:12.647176 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:44:12.648021 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d897c53-8758-41aa-b980-346fc010eb90 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:44:12.684455 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14ae2a73-f46c-41a5-8674-6175a363d000 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:44:12.685116 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14ae2a73-f46c-41a5-8674-6175a363d000 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:44:12.685394 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14ae2a73-f46c-41a5-8674-6175a363d000 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:44:12.685690 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14ae2a73-f46c-41a5-8674-6175a363d000 None admin] PATH_INFO: `/v3/services` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:44:12.686312 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14ae2a73-f46c-41a5-8674-6175a363d000 None admin] RBAC: Adding query filter params (name=compute) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:44:12.686609 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14ae2a73-f46c-41a5-8674-6175a363d000 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:44:12.687999 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14ae2a73-f46c-41a5-8674-6175a363d000 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:44:12.693719 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 504/999] 10.222.0.27 () {56 vars in 1112 bytes} [Tue Aug 30 01:44:12 2022] GET /identity/v3/services?name=compute => generated 124 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:44:12.699447 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14ae2a73-f46c-41a5-8674-6175a363d000 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:44:12.705487 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 496/1000] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Aug 30 01:44:12 2022] POST /identity/v3/auth/tokens => generated 1570 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Aug 30 01:44:12.712641 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:44:12.803347 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14ae2a73-f46c-41a5-8674-6175a363d000 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:44:12.837154 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49ba65d2-90e5-4723-872e-eff6dbe2d72d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:44:12.837782 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49ba65d2-90e5-4723-872e-eff6dbe2d72d None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:44:12.838059 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49ba65d2-90e5-4723-872e-eff6dbe2d72d None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:44:12.838327 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49ba65d2-90e5-4723-872e-eff6dbe2d72d None admin] PATH_INFO: `/v3/services` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:44:12.838945 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49ba65d2-90e5-4723-872e-eff6dbe2d72d None admin] RBAC: Adding query filter params (type=compute) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:44:12.839242 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49ba65d2-90e5-4723-872e-eff6dbe2d72d None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:44:12.840620 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49ba65d2-90e5-4723-872e-eff6dbe2d72d None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:44:12.845477 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 505/1001] 10.222.0.27 () {56 vars in 1112 bytes} [Tue Aug 30 01:44:12 2022] GET /identity/v3/services?type=compute => generated 352 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:44:12.851700 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7747be3-2f28-4952-8bed-9b2c74cd4ec4 None None] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:44:12.953856 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7747be3-2f28-4952-8bed-9b2c74cd4ec4 None None] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:44:12.969247 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49ba65d2-90e5-4723-872e-eff6dbe2d72d None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:44:12.995920 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdea0629-e685-4158-bd57-1d2c5978dc90 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:44:12.996538 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdea0629-e685-4158-bd57-1d2c5978dc90 None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:44:12.997333 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdea0629-e685-4158-bd57-1d2c5978dc90 None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:44:12.997641 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdea0629-e685-4158-bd57-1d2c5978dc90 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:44:12.998632 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bdea0629-e685-4158-bd57-1d2c5978dc90 None admin] RBAC: Adding query filter params (interface=public, service_id=3b4e7762a49d425babcf06f0efa09f00, region_id=RegionOne) {{(pid=81578) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:44:12.998941 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bdea0629-e685-4158-bd57-1d2c5978dc90 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:44:13.000779 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bdea0629-e685-4158-bd57-1d2c5978dc90 None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:44:13.009609 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 497/1002] 10.222.0.27 () {56 vars in 1251 bytes} [Tue Aug 30 01:44:12 2022] GET /identity/v3/endpoints?service_id=3b4e7762a49d425babcf06f0efa09f00&interface=public®ion_id=RegionOne => generated 194 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 30 01:44:13.014722 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdea0629-e685-4158-bd57-1d2c5978dc90 None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:44:13.040048 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:44:13.058294 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:44:13.083944 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49ba65d2-90e5-4723-872e-eff6dbe2d72d None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:44:13.118249 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdea0629-e685-4158-bd57-1d2c5978dc90 None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:44:13.120859 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c7a6d6a-e113-4054-b53e-13578f661974 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:44:13.121527 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c7a6d6a-e113-4054-b53e-13578f661974 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:44:13.121812 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c7a6d6a-e113-4054-b53e-13578f661974 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:44:13.122080 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c7a6d6a-e113-4054-b53e-13578f661974 None admin] PATH_INFO: `/v3/services/object-store` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:44:13.122651 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c7a6d6a-e113-4054-b53e-13578f661974 None admin] RBAC: Authorizing `identity:get_service(service_id=object-store)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:44:13.124001 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c7a6d6a-e113-4054-b53e-13578f661974 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:44:13.133366 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application [None req-1c7a6d6a-e113-4054-b53e-13578f661974 None admin] Could not find service: object-store.: keystone.exception.ServiceNotFound: Could not find service: object-store. Aug 30 01:44:13.133366 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 01:44:13.133366 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Aug 30 01:44:13.133366 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Aug 30 01:44:13.133366 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Aug 30 01:44:13.133366 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Aug 30 01:44:13.133366 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Aug 30 01:44:13.133366 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 30 01:44:13.133366 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Aug 30 01:44:13.133366 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Aug 30 01:44:13.133366 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 30 01:44:13.133366 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Aug 30 01:44:13.133366 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 01:44:13.133366 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 01:44:13.133366 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 01:44:13.133366 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 01:44:13.133366 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 01:44:13.134597 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 01:44:13.134597 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 01:44:13.134597 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 01:44:13.134597 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 01:44:13.134597 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 01:44:13.134597 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 01:44:13.134597 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Aug 30 01:44:13.134597 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_service(service_id) Aug 30 01:44:13.134597 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Aug 30 01:44:13.134597 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Aug 30 01:44:13.134597 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 01:44:13.134597 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 01:44:13.134597 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 01:44:13.134597 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 01:44:13.134597 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 01:44:13.134597 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 01:44:13.135453 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 01:44:13.135453 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application with Lock( Aug 30 01:44:13.135453 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 01:44:13.135453 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._enter() Aug 30 01:44:13.135453 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 01:44:13.135453 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 01:44:13.135453 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 01:44:13.135453 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.creator() Aug 30 01:44:13.135453 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 01:44:13.135453 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application created_value = creator( Aug 30 01:44:13.135453 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Aug 30 01:44:13.135453 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 30 01:44:13.135453 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Aug 30 01:44:13.135453 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Aug 30 01:44:13.135453 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 506/1003] 10.222.0.27 () {56 vars in 1126 bytes} [Tue Aug 30 01:44:12 2022] GET /identity/v3/services/object-store => generated 93 bytes in 166 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 30 01:44:13.140269 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c7a6d6a-e113-4054-b53e-13578f661974 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:44:13.156099 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a3a4bb5-e786-43d0-820b-1bc7d21a1adb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:44:13.156720 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a3a4bb5-e786-43d0-820b-1bc7d21a1adb None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:44:13.156995 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a3a4bb5-e786-43d0-820b-1bc7d21a1adb None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:44:13.157273 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a3a4bb5-e786-43d0-820b-1bc7d21a1adb None admin] PATH_INFO: `/v3/services` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:44:13.157896 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a3a4bb5-e786-43d0-820b-1bc7d21a1adb None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:44:13.159237 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a3a4bb5-e786-43d0-820b-1bc7d21a1adb None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:44:13.166156 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 498/1004] 10.222.0.27 () {56 vars in 1087 bytes} [Tue Aug 30 01:44:13 2022] GET /identity/v3/services => generated 1727 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 30 01:44:13.252755 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c7a6d6a-e113-4054-b53e-13578f661974 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:44:13.290471 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef8934c2-828f-4a1e-bb62-bc8692fe1411 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:44:13.291188 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef8934c2-828f-4a1e-bb62-bc8692fe1411 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:44:13.291537 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef8934c2-828f-4a1e-bb62-bc8692fe1411 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:44:13.291874 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef8934c2-828f-4a1e-bb62-bc8692fe1411 None admin] PATH_INFO: `/v3/services` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:44:13.292593 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef8934c2-828f-4a1e-bb62-bc8692fe1411 None admin] RBAC: Adding query filter params (name=object-store) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:44:13.292968 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef8934c2-828f-4a1e-bb62-bc8692fe1411 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:44:13.294566 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef8934c2-828f-4a1e-bb62-bc8692fe1411 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:44:13.300936 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 507/1005] 10.222.0.27 () {56 vars in 1122 bytes} [Tue Aug 30 01:44:13 2022] GET /identity/v3/services?name=object-store => generated 129 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 30 01:44:13.306220 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a3a4bb5-e786-43d0-820b-1bc7d21a1adb None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:44:13.399590 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:44:13.414951 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a3a4bb5-e786-43d0-820b-1bc7d21a1adb None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:44:13.449200 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-838f576e-79eb-44ea-bb20-9f9e94fbf98b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:44:13.449828 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-838f576e-79eb-44ea-bb20-9f9e94fbf98b None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:44:13.450106 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-838f576e-79eb-44ea-bb20-9f9e94fbf98b None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:44:13.450373 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-838f576e-79eb-44ea-bb20-9f9e94fbf98b None admin] PATH_INFO: `/v3/services` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:44:13.450994 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-838f576e-79eb-44ea-bb20-9f9e94fbf98b None admin] RBAC: Adding query filter params (type=object-store) {{(pid=81578) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:44:13.451290 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-838f576e-79eb-44ea-bb20-9f9e94fbf98b None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:44:13.452636 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-838f576e-79eb-44ea-bb20-9f9e94fbf98b None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:44:13.462194 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 499/1006] 10.222.0.27 () {56 vars in 1122 bytes} [Tue Aug 30 01:44:13 2022] GET /identity/v3/services?type=object-store => generated 356 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:44:13.463697 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef8934c2-828f-4a1e-bb62-bc8692fe1411 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:44:13.466793 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:44:13.566416 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef8934c2-828f-4a1e-bb62-bc8692fe1411 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:44:13.605643 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2a1975c-3e09-4147-9b54-2eaccdea5bb5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:44:13.606611 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2a1975c-3e09-4147-9b54-2eaccdea5bb5 None admin] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:44:13.606893 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2a1975c-3e09-4147-9b54-2eaccdea5bb5 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:44:13.607159 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2a1975c-3e09-4147-9b54-2eaccdea5bb5 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:44:13.607849 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2a1975c-3e09-4147-9b54-2eaccdea5bb5 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:44:13.609215 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2a1975c-3e09-4147-9b54-2eaccdea5bb5 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:44:13.687684 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 508/1007] 10.222.0.27 () {60 vars in 1144 bytes} [Tue Aug 30 01:44:13 2022] POST /identity/v3/endpoints => generated 325 bytes in 224 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 01:44:13.864727 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c372762-12e6-4624-b01b-b0460ef4a317 None None] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:44:13.865207 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c372762-12e6-4624-b01b-b0460ef4a317 None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:44:13.865207 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c372762-12e6-4624-b01b-b0460ef4a317 None None] PATH_INFO: `/` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:44:13.866520 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 500/1008] 10.222.0.27 () {54 vars in 927 bytes} [Tue Aug 30 01:44:13 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 01:44:13.871727 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bb1cd9c-ba72-4403-90eb-83265d77c37f None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:44:13.872022 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bb1cd9c-ba72-4403-90eb-83265d77c37f None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:44:13.872303 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bb1cd9c-ba72-4403-90eb-83265d77c37f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:44:13.891295 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-7bb1cd9c-ba72-4403-90eb-83265d77c37f None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 01:44:13.892493 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-7bb1cd9c-ba72-4403-90eb-83265d77c37f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:44:13.951614 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:44:13.973221 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 509/1009] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Aug 30 01:44:13 2022] POST /identity/v3/auth/tokens => generated 1717 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 30 01:44:14.248742 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c372762-12e6-4624-b01b-b0460ef4a317 None None] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:44:14.362292 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c372762-12e6-4624-b01b-b0460ef4a317 None None] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:44:14.396887 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f2dc165-25a9-4c6a-bd48-b59733bf61df None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:44:14.397513 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f2dc165-25a9-4c6a-bd48-b59733bf61df None admin] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:44:14.397795 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f2dc165-25a9-4c6a-bd48-b59733bf61df None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:44:14.398065 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f2dc165-25a9-4c6a-bd48-b59733bf61df None admin] PATH_INFO: `/v3/services` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:44:14.398709 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f2dc165-25a9-4c6a-bd48-b59733bf61df None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:44:14.400060 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f2dc165-25a9-4c6a-bd48-b59733bf61df None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:44:14.492256 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 501/1010] 10.222.0.27 () {60 vars in 1141 bytes} [Tue Aug 30 01:44:14 2022] POST /identity/v3/services => generated 242 bytes in 244 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 01:44:15.091484 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-287153dd-11d0-4a3d-b340-b33680e0e71e None None] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:44:15.092029 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-287153dd-11d0-4a3d-b340-b33680e0e71e None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:44:15.092029 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-287153dd-11d0-4a3d-b340-b33680e0e71e None None] PATH_INFO: `/` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:44:15.093428 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 510/1011] 10.222.0.27 () {54 vars in 927 bytes} [Tue Aug 30 01:44:15 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 01:44:15.098530 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea902e5b-9230-4e87-b7ce-71c4e6c7c408 None None] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:44:15.098926 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea902e5b-9230-4e87-b7ce-71c4e6c7c408 None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:44:15.099090 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea902e5b-9230-4e87-b7ce-71c4e6c7c408 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:44:15.117372 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.auth.core [None req-ea902e5b-9230-4e87-b7ce-71c4e6c7c408 None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81578) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 01:44:15.118598 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-ea902e5b-9230-4e87-b7ce-71c4e6c7c408 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:44:15.122380 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:44:15.194569 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 502/1012] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Aug 30 01:44:15 2022] POST /identity/v3/auth/tokens => generated 1813 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 30 01:44:15.463489 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-287153dd-11d0-4a3d-b340-b33680e0e71e None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:44:15.566938 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-287153dd-11d0-4a3d-b340-b33680e0e71e None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:44:15.602042 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f899ac6a-da6f-44f6-b266-c26b33e87c52 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:44:15.602676 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f899ac6a-da6f-44f6-b266-c26b33e87c52 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:44:15.603005 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f899ac6a-da6f-44f6-b266-c26b33e87c52 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:44:15.603277 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f899ac6a-da6f-44f6-b266-c26b33e87c52 None admin] PATH_INFO: `/v3/services/compute` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:44:15.603862 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f899ac6a-da6f-44f6-b266-c26b33e87c52 None admin] RBAC: Authorizing `identity:get_service(service_id=compute)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:44:15.605225 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f899ac6a-da6f-44f6-b266-c26b33e87c52 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:44:15.615232 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application [None req-f899ac6a-da6f-44f6-b266-c26b33e87c52 None admin] Could not find service: compute.: keystone.exception.ServiceNotFound: Could not find service: compute. Aug 30 01:44:15.615232 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 01:44:15.615232 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Aug 30 01:44:15.615232 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Aug 30 01:44:15.615232 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Aug 30 01:44:15.615232 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Aug 30 01:44:15.615232 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Aug 30 01:44:15.615232 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 30 01:44:15.615232 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Aug 30 01:44:15.615232 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Aug 30 01:44:15.615232 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 30 01:44:15.615232 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Aug 30 01:44:15.615232 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 01:44:15.615232 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 01:44:15.615232 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 01:44:15.615232 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 01:44:15.615232 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 01:44:15.616337 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 01:44:15.616337 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 01:44:15.616337 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 01:44:15.616337 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 01:44:15.616337 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 01:44:15.616337 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 01:44:15.616337 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Aug 30 01:44:15.616337 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_service(service_id) Aug 30 01:44:15.616337 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Aug 30 01:44:15.616337 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Aug 30 01:44:15.616337 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 01:44:15.616337 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 01:44:15.616337 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 01:44:15.616337 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 01:44:15.616337 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 01:44:15.616337 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 01:44:15.617204 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 01:44:15.617204 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application with Lock( Aug 30 01:44:15.617204 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 01:44:15.617204 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._enter() Aug 30 01:44:15.617204 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 01:44:15.617204 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 01:44:15.617204 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 01:44:15.617204 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.creator() Aug 30 01:44:15.617204 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 01:44:15.617204 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application created_value = creator( Aug 30 01:44:15.617204 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Aug 30 01:44:15.617204 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 30 01:44:15.617204 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Aug 30 01:44:15.617204 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Aug 30 01:44:15.621275 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 511/1013] 10.222.0.27 () {56 vars in 1111 bytes} [Tue Aug 30 01:44:15 2022] GET /identity/v3/services/compute => generated 88 bytes in 158 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 01:44:15.622808 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea902e5b-9230-4e87-b7ce-71c4e6c7c408 None None] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:44:15.700436 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d0436a9-89e1-4f1a-9b2e-b07a21c59d77 None None] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:44:15.700759 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d0436a9-89e1-4f1a-9b2e-b07a21c59d77 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:44:15.701134 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d0436a9-89e1-4f1a-9b2e-b07a21c59d77 None None] PATH_INFO: `/` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:44:15.702727 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 512/1014] 10.222.0.27 () {54 vars in 927 bytes} [Tue Aug 30 01:44:15 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 30 01:44:15.708623 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f86feafe-0c51-4d72-9c12-ec7a7ca9c9b2 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:44:15.708897 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f86feafe-0c51-4d72-9c12-ec7a7ca9c9b2 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:44:15.710886 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f86feafe-0c51-4d72-9c12-ec7a7ca9c9b2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:44:15.731550 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-f86feafe-0c51-4d72-9c12-ec7a7ca9c9b2 None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 01:44:15.732737 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-f86feafe-0c51-4d72-9c12-ec7a7ca9c9b2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:44:15.735988 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea902e5b-9230-4e87-b7ce-71c4e6c7c408 None None] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:44:15.770724 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-490fe21b-05fb-43da-bfdb-9e0968ffaa5b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:44:15.771320 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-490fe21b-05fb-43da-bfdb-9e0968ffaa5b None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:44:15.771594 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-490fe21b-05fb-43da-bfdb-9e0968ffaa5b None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:44:15.771857 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-490fe21b-05fb-43da-bfdb-9e0968ffaa5b None admin] PATH_INFO: `/v3/services` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:44:15.772477 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-490fe21b-05fb-43da-bfdb-9e0968ffaa5b None admin] RBAC: Adding query filter params (name=compute) {{(pid=81578) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:44:15.772776 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-490fe21b-05fb-43da-bfdb-9e0968ffaa5b None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:44:15.774166 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-490fe21b-05fb-43da-bfdb-9e0968ffaa5b None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:44:15.775898 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:44:15.781003 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 503/1015] 10.222.0.27 () {56 vars in 1112 bytes} [Tue Aug 30 01:44:15 2022] GET /identity/v3/services?name=compute => generated 124 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:44:15.787352 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-490fe21b-05fb-43da-bfdb-9e0968ffaa5b None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:44:15.794449 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:44:15.809195 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 513/1016] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Aug 30 01:44:15 2022] POST /identity/v3/auth/tokens => generated 1813 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Aug 30 01:44:15.887054 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-490fe21b-05fb-43da-bfdb-9e0968ffaa5b None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:44:15.920758 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c1b3b9e-5c5e-4f3f-8f7e-11695fa4f717 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:44:15.921362 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c1b3b9e-5c5e-4f3f-8f7e-11695fa4f717 None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:44:15.921665 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c1b3b9e-5c5e-4f3f-8f7e-11695fa4f717 None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:44:15.921937 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c1b3b9e-5c5e-4f3f-8f7e-11695fa4f717 None admin] PATH_INFO: `/v3/services` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:44:15.922568 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c1b3b9e-5c5e-4f3f-8f7e-11695fa4f717 None admin] RBAC: Adding query filter params (type=compute) {{(pid=81578) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:44:15.922870 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c1b3b9e-5c5e-4f3f-8f7e-11695fa4f717 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:44:15.924216 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c1b3b9e-5c5e-4f3f-8f7e-11695fa4f717 None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:44:15.929714 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 504/1017] 10.222.0.27 () {56 vars in 1112 bytes} [Tue Aug 30 01:44:15 2022] GET /identity/v3/services?type=compute => generated 352 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 30 01:44:15.935296 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f86feafe-0c51-4d72-9c12-ec7a7ca9c9b2 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:44:16.037773 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f86feafe-0c51-4d72-9c12-ec7a7ca9c9b2 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:44:16.072565 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb7fb4ee-63bc-4083-84f2-08f9168278ca None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:44:16.073167 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb7fb4ee-63bc-4083-84f2-08f9168278ca None admin] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:44:16.073445 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb7fb4ee-63bc-4083-84f2-08f9168278ca None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:44:16.073741 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb7fb4ee-63bc-4083-84f2-08f9168278ca None admin] PATH_INFO: `/v3/endpoints` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:44:16.074415 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb7fb4ee-63bc-4083-84f2-08f9168278ca None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:44:16.075787 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb7fb4ee-63bc-4083-84f2-08f9168278ca None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:44:16.079142 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c1b3b9e-5c5e-4f3f-8f7e-11695fa4f717 None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:44:16.141862 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:44:16.180217 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c1b3b9e-5c5e-4f3f-8f7e-11695fa4f717 None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:44:16.201487 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 514/1018] 10.222.0.27 () {60 vars in 1144 bytes} [Tue Aug 30 01:44:15 2022] POST /identity/v3/endpoints => generated 334 bytes in 256 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 01:44:16.235775 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0679641-ca52-4dd4-bfba-318ed57c655f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:44:16.236414 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0679641-ca52-4dd4-bfba-318ed57c655f None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:44:16.236692 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0679641-ca52-4dd4-bfba-318ed57c655f None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:44:16.236956 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0679641-ca52-4dd4-bfba-318ed57c655f None admin] PATH_INFO: `/v3/domains/default` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:44:16.238707 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0679641-ca52-4dd4-bfba-318ed57c655f None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:44:16.240285 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0679641-ca52-4dd4-bfba-318ed57c655f None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:44:16.242207 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 505/1019] 10.222.0.27 () {56 vars in 1108 bytes} [Tue Aug 30 01:44:16 2022] GET /identity/v3/domains/default => generated 199 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 30 01:44:16.248487 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb7fb4ee-63bc-4083-84f2-08f9168278ca None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:44:16.297957 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:44:16.365788 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb7fb4ee-63bc-4083-84f2-08f9168278ca None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:44:16.400568 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b0fae95-9978-4ef0-84b4-18f6c5d9c198 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:44:16.401175 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b0fae95-9978-4ef0-84b4-18f6c5d9c198 None admin] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:44:16.401509 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b0fae95-9978-4ef0-84b4-18f6c5d9c198 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:44:16.401742 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b0fae95-9978-4ef0-84b4-18f6c5d9c198 None admin] PATH_INFO: `/v3/projects` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:44:16.402400 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b0fae95-9978-4ef0-84b4-18f6c5d9c198 None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:44:16.403808 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b0fae95-9978-4ef0-84b4-18f6c5d9c198 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:44:16.493520 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d1a4d19-f8a3-41e4-bcf5-6974d8ff5893 None None] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:44:16.493774 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d1a4d19-f8a3-41e4-bcf5-6974d8ff5893 None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:44:16.494055 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d1a4d19-f8a3-41e4-bcf5-6974d8ff5893 None None] PATH_INFO: `/` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:44:16.494418 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 515/1020] 10.222.0.27 () {60 vars in 1141 bytes} [Tue Aug 30 01:44:16 2022] POST /identity/v3/projects => generated 311 bytes in 246 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 01:44:16.495376 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 506/1021] 10.222.0.27 () {54 vars in 927 bytes} [Tue Aug 30 01:44:16 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 30 01:44:16.501013 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a40be21-0408-4bdb-8a80-ca5d221fd73d None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:44:16.501302 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a40be21-0408-4bdb-8a80-ca5d221fd73d None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:44:16.501656 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a40be21-0408-4bdb-8a80-ca5d221fd73d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:44:16.522732 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-5a40be21-0408-4bdb-8a80-ca5d221fd73d None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 01:44:16.523952 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-5a40be21-0408-4bdb-8a80-ca5d221fd73d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:44:16.600602 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 516/1022] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Aug 30 01:44:16 2022] POST /identity/v3/auth/tokens => generated 1969 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 30 01:44:16.893045 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d1a4d19-f8a3-41e4-bcf5-6974d8ff5893 None None] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:44:17.002339 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d1a4d19-f8a3-41e4-bcf5-6974d8ff5893 None None] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:44:17.030894 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:44:17.040014 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4058bece-793a-45c9-b109-9c78a3e1478d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:44:17.040643 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4058bece-793a-45c9-b109-9c78a3e1478d None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:44:17.040926 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4058bece-793a-45c9-b109-9c78a3e1478d None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:44:17.041198 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4058bece-793a-45c9-b109-9c78a3e1478d None admin] PATH_INFO: `/v3/services/image` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:44:17.041806 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4058bece-793a-45c9-b109-9c78a3e1478d None admin] RBAC: Authorizing `identity:get_service(service_id=image)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:44:17.043300 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4058bece-793a-45c9-b109-9c78a3e1478d None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:44:17.053577 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application [None req-4058bece-793a-45c9-b109-9c78a3e1478d None admin] Could not find service: image.: keystone.exception.ServiceNotFound: Could not find service: image. Aug 30 01:44:17.053577 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 01:44:17.053577 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Aug 30 01:44:17.053577 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Aug 30 01:44:17.053577 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Aug 30 01:44:17.053577 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Aug 30 01:44:17.053577 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Aug 30 01:44:17.053577 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 30 01:44:17.053577 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Aug 30 01:44:17.053577 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application Aug 30 01:44:17.053577 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 30 01:44:17.053577 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application Aug 30 01:44:17.053577 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 01:44:17.053577 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 01:44:17.053577 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 01:44:17.053577 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 01:44:17.053577 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 01:44:17.055976 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 01:44:17.055976 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 01:44:17.055976 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 01:44:17.055976 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 01:44:17.055976 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 01:44:17.055976 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 01:44:17.055976 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Aug 30 01:44:17.055976 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._get_service(service_id) Aug 30 01:44:17.055976 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Aug 30 01:44:17.055976 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Aug 30 01:44:17.055976 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 01:44:17.055976 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 01:44:17.055976 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 01:44:17.055976 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 01:44:17.055976 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 01:44:17.055976 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 01:44:17.056814 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 01:44:17.056814 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application with Lock( Aug 30 01:44:17.056814 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 01:44:17.056814 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._enter() Aug 30 01:44:17.056814 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 01:44:17.056814 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 01:44:17.056814 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 01:44:17.056814 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.creator() Aug 30 01:44:17.056814 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 01:44:17.056814 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application created_value = creator( Aug 30 01:44:17.056814 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Aug 30 01:44:17.056814 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 30 01:44:17.056814 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Aug 30 01:44:17.056814 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application Aug 30 01:44:17.056814 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 507/1023] 10.222.0.27 () {56 vars in 1105 bytes} [Tue Aug 30 01:44:16 2022] GET /identity/v3/services/image => generated 86 bytes in 162 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 01:44:17.060612 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a40be21-0408-4bdb-8a80-ca5d221fd73d None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:44:17.082988 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:44:17.164448 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a40be21-0408-4bdb-8a80-ca5d221fd73d None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:44:17.199737 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa720bca-2695-40a4-a29e-5930f44a6736 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:44:17.200346 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa720bca-2695-40a4-a29e-5930f44a6736 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:44:17.200625 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa720bca-2695-40a4-a29e-5930f44a6736 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:44:17.200896 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa720bca-2695-40a4-a29e-5930f44a6736 None admin] PATH_INFO: `/v3/services` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:44:17.201553 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa720bca-2695-40a4-a29e-5930f44a6736 None admin] RBAC: Adding query filter params (name=image) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:44:17.202094 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa720bca-2695-40a4-a29e-5930f44a6736 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:44:17.203213 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa720bca-2695-40a4-a29e-5930f44a6736 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:44:17.208783 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 517/1024] 10.222.0.27 () {56 vars in 1108 bytes} [Tue Aug 30 01:44:17 2022] GET /identity/v3/services?name=image => generated 122 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:44:17.214256 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4058bece-793a-45c9-b109-9c78a3e1478d None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:44:17.317008 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4058bece-793a-45c9-b109-9c78a3e1478d None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:44:17.354172 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34e98e0d-526e-4f96-bccd-1e472fb421f6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:44:17.354779 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34e98e0d-526e-4f96-bccd-1e472fb421f6 None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:44:17.355060 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34e98e0d-526e-4f96-bccd-1e472fb421f6 None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:44:17.355331 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34e98e0d-526e-4f96-bccd-1e472fb421f6 None admin] PATH_INFO: `/v3/services` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:44:17.355965 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34e98e0d-526e-4f96-bccd-1e472fb421f6 None admin] RBAC: Adding query filter params (type=image) {{(pid=81578) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:44:17.356269 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34e98e0d-526e-4f96-bccd-1e472fb421f6 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:44:17.357666 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34e98e0d-526e-4f96-bccd-1e472fb421f6 None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:44:17.368181 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 508/1025] 10.222.0.27 () {56 vars in 1108 bytes} [Tue Aug 30 01:44:17 2022] GET /identity/v3/services?type=image => generated 350 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 30 01:44:17.370343 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa720bca-2695-40a4-a29e-5930f44a6736 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:44:17.462068 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:44:17.479384 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa720bca-2695-40a4-a29e-5930f44a6736 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:44:17.514814 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ad6b22b-07c3-4322-8d9e-692fc3ed9319 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:44:17.515420 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ad6b22b-07c3-4322-8d9e-692fc3ed9319 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:44:17.515702 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ad6b22b-07c3-4322-8d9e-692fc3ed9319 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:44:17.515974 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ad6b22b-07c3-4322-8d9e-692fc3ed9319 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:44:17.516674 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ad6b22b-07c3-4322-8d9e-692fc3ed9319 None admin] RBAC: Adding query filter params (interface=public, service_id=8e1d011e6c054f3cbf4120b8aab8e8c3, region_id=RegionOne) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:44:17.516983 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ad6b22b-07c3-4322-8d9e-692fc3ed9319 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:44:17.518377 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ad6b22b-07c3-4322-8d9e-692fc3ed9319 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:44:17.528813 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 518/1026] 10.222.0.27 () {56 vars in 1251 bytes} [Tue Aug 30 01:44:17 2022] GET /identity/v3/endpoints?service_id=8e1d011e6c054f3cbf4120b8aab8e8c3&interface=public®ion_id=RegionOne => generated 194 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:44:17.530362 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34e98e0d-526e-4f96-bccd-1e472fb421f6 None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:44:17.589927 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:44:17.637837 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34e98e0d-526e-4f96-bccd-1e472fb421f6 None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:44:17.672849 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86c94dce-f833-400e-8b1c-f3e3f9b5429c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:44:17.673718 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86c94dce-f833-400e-8b1c-f3e3f9b5429c None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:44:17.673992 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86c94dce-f833-400e-8b1c-f3e3f9b5429c None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:44:17.674259 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86c94dce-f833-400e-8b1c-f3e3f9b5429c None admin] PATH_INFO: `/v3/services` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:44:17.674860 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86c94dce-f833-400e-8b1c-f3e3f9b5429c None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:44:17.676221 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86c94dce-f833-400e-8b1c-f3e3f9b5429c None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:44:17.684545 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 509/1027] 10.222.0.27 () {56 vars in 1087 bytes} [Tue Aug 30 01:44:17 2022] GET /identity/v3/services => generated 1957 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 01:44:18.237474 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e3cb5c6-8493-49bb-89ef-4ee9d0fd3ad4 None None] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:44:18.238528 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e3cb5c6-8493-49bb-89ef-4ee9d0fd3ad4 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:44:18.238635 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e3cb5c6-8493-49bb-89ef-4ee9d0fd3ad4 None None] PATH_INFO: `/` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:44:18.240117 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 519/1028] 10.222.0.27 () {54 vars in 927 bytes} [Tue Aug 30 01:44:18 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 01:44:18.245083 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26086e63-594c-44f4-9112-6e9492fb5a87 None None] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:44:18.245526 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26086e63-594c-44f4-9112-6e9492fb5a87 None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:44:18.245706 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26086e63-594c-44f4-9112-6e9492fb5a87 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:44:18.264269 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.auth.core [None req-26086e63-594c-44f4-9112-6e9492fb5a87 None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81578) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 01:44:18.265435 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-26086e63-594c-44f4-9112-6e9492fb5a87 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:44:18.337522 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 510/1029] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Aug 30 01:44:18 2022] POST /identity/v3/auth/tokens => generated 1969 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 30 01:44:18.471987 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71c00f36-ee29-4cee-81c6-2a4ae0a46017 None None] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:44:18.472414 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71c00f36-ee29-4cee-81c6-2a4ae0a46017 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:44:18.473351 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71c00f36-ee29-4cee-81c6-2a4ae0a46017 None None] PATH_INFO: `/` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:44:18.474867 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 520/1030] 10.222.0.27 () {54 vars in 927 bytes} [Tue Aug 30 01:44:18 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 01:44:18.480299 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e78b5d31-1e1d-4bc9-ae9a-0043843c7857 None None] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:44:18.480596 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e78b5d31-1e1d-4bc9-ae9a-0043843c7857 None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:44:18.480875 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e78b5d31-1e1d-4bc9-ae9a-0043843c7857 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:44:18.499154 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.auth.core [None req-e78b5d31-1e1d-4bc9-ae9a-0043843c7857 None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81578) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 01:44:18.500272 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-e78b5d31-1e1d-4bc9-ae9a-0043843c7857 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:44:18.547710 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:44:18.568699 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 511/1031] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Aug 30 01:44:18 2022] POST /identity/v3/auth/tokens => generated 1969 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 30 01:44:18.610360 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71c00f36-ee29-4cee-81c6-2a4ae0a46017 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:44:18.724010 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71c00f36-ee29-4cee-81c6-2a4ae0a46017 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:44:18.761958 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e072832-ce46-4d2b-85f5-1c414d1c0433 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:44:18.762512 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e072832-ce46-4d2b-85f5-1c414d1c0433 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:44:18.762791 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e072832-ce46-4d2b-85f5-1c414d1c0433 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:44:18.763061 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e072832-ce46-4d2b-85f5-1c414d1c0433 None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:44:18.774346 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e072832-ce46-4d2b-85f5-1c414d1c0433 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 30 01:44:18.779266 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application [None req-0e072832-ce46-4d2b-85f5-1c414d1c0433 None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Aug 30 01:44:18.779266 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 01:44:18.779266 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 01:44:18.779266 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 01:44:18.779266 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 01:44:18.779266 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 01:44:18.779266 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 01:44:18.779266 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 01:44:18.779266 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 01:44:18.779266 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 01:44:18.779266 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 01:44:18.779266 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 01:44:18.779266 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 30 01:44:18.779266 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 30 01:44:18.779266 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 30 01:44:18.779266 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 30 01:44:18.780207 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 30 01:44:18.780207 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 30 01:44:18.780207 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 30 01:44:18.780207 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 30 01:44:18.780207 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 01:44:18.780207 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 01:44:18.780207 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 01:44:18.780207 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 01:44:18.780207 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 01:44:18.780207 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 01:44:18.780207 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 01:44:18.780207 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application with Lock( Aug 30 01:44:18.780207 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 01:44:18.780207 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._enter() Aug 30 01:44:18.780207 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 01:44:18.780207 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 01:44:18.781091 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 01:44:18.781091 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.creator() Aug 30 01:44:18.781091 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 01:44:18.781091 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application created_value = creator( Aug 30 01:44:18.781091 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 30 01:44:18.781091 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 30 01:44:18.781091 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 30 01:44:18.781091 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 30 01:44:18.781091 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 30 01:44:18.781091 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 30 01:44:18.781091 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Aug 30 01:44:18.781091 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Aug 30 01:44:18.785719 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 521/1032] 10.222.0.27 () {56 vars in 1120 bytes} [Tue Aug 30 01:44:18 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 175 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 01:44:18.786499 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e78b5d31-1e1d-4bc9-ae9a-0043843c7857 None None] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:44:18.848977 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e072832-ce46-4d2b-85f5-1c414d1c0433 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:44:18.863724 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:44:18.888893 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e78b5d31-1e1d-4bc9-ae9a-0043843c7857 None None] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:44:18.926809 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73a4389b-c882-4f2c-b511-04680e9b9a6f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:44:18.927558 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73a4389b-c882-4f2c-b511-04680e9b9a6f None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:44:18.927695 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73a4389b-c882-4f2c-b511-04680e9b9a6f None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:44:18.927961 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73a4389b-c882-4f2c-b511-04680e9b9a6f None admin] PATH_INFO: `/v3/roles` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:44:18.928643 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73a4389b-c882-4f2c-b511-04680e9b9a6f None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=81578) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:44:18.928925 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73a4389b-c882-4f2c-b511-04680e9b9a6f None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:44:18.930315 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73a4389b-c882-4f2c-b511-04680e9b9a6f None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:44:18.938694 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 512/1033] 10.222.0.27 () {56 vars in 1115 bytes} [Tue Aug 30 01:44:18 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 30 01:44:18.944510 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73a4389b-c882-4f2c-b511-04680e9b9a6f None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:44:18.964675 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e072832-ce46-4d2b-85f5-1c414d1c0433 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:44:18.993322 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:44:19.005324 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5865ddd-05a1-4f4e-929c-421c62f6673f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:44:19.006630 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5865ddd-05a1-4f4e-929c-421c62f6673f None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:44:19.006630 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5865ddd-05a1-4f4e-929c-421c62f6673f None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:44:19.006630 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5865ddd-05a1-4f4e-929c-421c62f6673f None admin] PATH_INFO: `/v3/domains/default` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:44:19.007834 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5865ddd-05a1-4f4e-929c-421c62f6673f None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:44:19.009280 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5865ddd-05a1-4f4e-929c-421c62f6673f None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:44:19.010923 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 522/1034] 10.222.0.27 () {56 vars in 1108 bytes} [Tue Aug 30 01:44:18 2022] GET /identity/v3/domains/default => generated 199 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:44:19.016666 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5865ddd-05a1-4f4e-929c-421c62f6673f None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:44:19.056451 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73a4389b-c882-4f2c-b511-04680e9b9a6f None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:44:19.096368 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eced00c3-b397-48e2-94e1-09f8001401f5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:44:19.097006 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eced00c3-b397-48e2-94e1-09f8001401f5 None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:44:19.097285 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eced00c3-b397-48e2-94e1-09f8001401f5 None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:44:19.097662 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eced00c3-b397-48e2-94e1-09f8001401f5 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:44:19.103575 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:44:19.110522 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eced00c3-b397-48e2-94e1-09f8001401f5 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:44:19.112511 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eced00c3-b397-48e2-94e1-09f8001401f5 None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:44:19.128113 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5865ddd-05a1-4f4e-929c-421c62f6673f None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:44:19.131832 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application [None req-eced00c3-b397-48e2-94e1-09f8001401f5 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 01:44:19.131832 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 01:44:19.131832 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 01:44:19.131832 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 01:44:19.131832 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 01:44:19.131832 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 01:44:19.131832 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 01:44:19.131832 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 01:44:19.131832 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 01:44:19.131832 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 01:44:19.131832 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 01:44:19.131832 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 01:44:19.131832 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 30 01:44:19.131832 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 30 01:44:19.131832 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 30 01:44:19.131832 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 30 01:44:19.132764 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 01:44:19.132764 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 01:44:19.132764 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 01:44:19.132764 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 01:44:19.132764 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 01:44:19.132764 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 01:44:19.132764 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 01:44:19.132764 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application with Lock( Aug 30 01:44:19.132764 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 01:44:19.132764 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._enter() Aug 30 01:44:19.132764 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 01:44:19.132764 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 01:44:19.132764 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 01:44:19.132764 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.creator() Aug 30 01:44:19.132764 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 01:44:19.132764 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application created_value = creator( Aug 30 01:44:19.132764 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 30 01:44:19.133711 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 30 01:44:19.133711 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 01:44:19.133711 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application Aug 30 01:44:19.133711 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 513/1035] 10.222.0.27 () {56 vars in 1108 bytes} [Tue Aug 30 01:44:18 2022] GET /identity/v3/domains/Default => generated 87 bytes in 189 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 01:44:19.138960 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eced00c3-b397-48e2-94e1-09f8001401f5 None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:44:19.170661 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05050e78-a39e-471a-a23f-7a96f2e37b63 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:44:19.171847 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05050e78-a39e-471a-a23f-7a96f2e37b63 None admin] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:44:19.172530 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05050e78-a39e-471a-a23f-7a96f2e37b63 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:44:19.173369 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05050e78-a39e-471a-a23f-7a96f2e37b63 None admin] PATH_INFO: `/v3/users` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:44:19.174502 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05050e78-a39e-471a-a23f-7a96f2e37b63 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:44:19.177354 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05050e78-a39e-471a-a23f-7a96f2e37b63 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:44:19.239416 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:44:19.246063 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eced00c3-b397-48e2-94e1-09f8001401f5 None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:44:19.258606 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 523/1036] 10.222.0.27 () {60 vars in 1132 bytes} [Tue Aug 30 01:44:19 2022] POST /identity/v3/users => generated 285 bytes in 242 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 30 01:44:19.284812 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46f8bea7-9bc5-4398-9ec9-0abdbd78e9be None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:44:19.286279 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46f8bea7-9bc5-4398-9ec9-0abdbd78e9be None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:44:19.286945 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46f8bea7-9bc5-4398-9ec9-0abdbd78e9be None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:44:19.287806 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46f8bea7-9bc5-4398-9ec9-0abdbd78e9be None admin] PATH_INFO: `/v3/domains` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:44:19.288476 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46f8bea7-9bc5-4398-9ec9-0abdbd78e9be None admin] RBAC: Adding query filter params (name=Default) {{(pid=81578) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:44:19.288826 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46f8bea7-9bc5-4398-9ec9-0abdbd78e9be None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:44:19.290322 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46f8bea7-9bc5-4398-9ec9-0abdbd78e9be None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:44:19.308710 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 514/1037] 10.222.0.27 () {56 vars in 1109 bytes} [Tue Aug 30 01:44:19 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:44:19.310043 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05050e78-a39e-471a-a23f-7a96f2e37b63 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:44:19.430928 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05050e78-a39e-471a-a23f-7a96f2e37b63 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:44:19.461249 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:44:19.466290 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a65beaf-93b8-43b5-b542-fc928728eb74 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:44:19.470569 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a65beaf-93b8-43b5-b542-fc928728eb74 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:44:19.470569 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a65beaf-93b8-43b5-b542-fc928728eb74 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:44:19.470569 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a65beaf-93b8-43b5-b542-fc928728eb74 None admin] PATH_INFO: `/v3/users/nova` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:44:19.476031 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a65beaf-93b8-43b5-b542-fc928728eb74 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:44:19.477773 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a65beaf-93b8-43b5-b542-fc928728eb74 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:44:19.490711 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application [None req-4a65beaf-93b8-43b5-b542-fc928728eb74 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Aug 30 01:44:19.490711 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 01:44:19.490711 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 01:44:19.490711 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 01:44:19.490711 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 01:44:19.490711 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 01:44:19.490711 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 01:44:19.490711 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 01:44:19.490711 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 01:44:19.490711 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 01:44:19.490711 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 01:44:19.490711 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 01:44:19.490711 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 30 01:44:19.490711 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 30 01:44:19.490711 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 30 01:44:19.490711 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 30 01:44:19.491729 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 01:44:19.491729 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 01:44:19.491729 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 30 01:44:19.491729 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 30 01:44:19.491729 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 30 01:44:19.491729 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 30 01:44:19.491729 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 01:44:19.491729 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 01:44:19.491729 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 01:44:19.491729 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 01:44:19.491729 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 01:44:19.491729 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application with Lock( Aug 30 01:44:19.491729 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 01:44:19.491729 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._enter() Aug 30 01:44:19.491729 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 01:44:19.491729 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 01:44:19.491729 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 01:44:19.491729 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.creator() Aug 30 01:44:19.492723 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 01:44:19.492723 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application created_value = creator( Aug 30 01:44:19.492723 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 30 01:44:19.492723 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 30 01:44:19.492723 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 30 01:44:19.492723 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 30 01:44:19.492723 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 30 01:44:19.492723 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 30 01:44:19.492723 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Aug 30 01:44:19.492723 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Aug 30 01:44:19.493504 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 524/1038] 10.222.0.27 () {56 vars in 1093 bytes} [Tue Aug 30 01:44:19 2022] GET /identity/v3/users/nova => generated 82 bytes in 184 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 01:44:19.499737 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46f8bea7-9bc5-4398-9ec9-0abdbd78e9be None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:44:19.589013 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f9ac0be-9d30-4866-8826-397fa354558f None None] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:44:19.589295 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f9ac0be-9d30-4866-8826-397fa354558f None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:44:19.589607 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f9ac0be-9d30-4866-8826-397fa354558f None None] PATH_INFO: `/` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:44:19.591093 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 525/1039] 10.222.0.27 () {54 vars in 927 bytes} [Tue Aug 30 01:44:19 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 01:44:19.595822 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88b42a3a-86b7-4470-b38c-e825f0e95c47 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:44:19.596110 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88b42a3a-86b7-4470-b38c-e825f0e95c47 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:44:19.596383 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88b42a3a-86b7-4470-b38c-e825f0e95c47 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:44:19.607120 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46f8bea7-9bc5-4398-9ec9-0abdbd78e9be None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:44:19.615966 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-88b42a3a-86b7-4470-b38c-e825f0e95c47 None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 01:44:19.617166 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-88b42a3a-86b7-4470-b38c-e825f0e95c47 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:44:19.646809 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffc57da7-a727-4066-9b57-9ca2b6a7aaa9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:44:19.647936 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffc57da7-a727-4066-9b57-9ca2b6a7aaa9 None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:44:19.648221 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffc57da7-a727-4066-9b57-9ca2b6a7aaa9 None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:44:19.648492 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffc57da7-a727-4066-9b57-9ca2b6a7aaa9 None admin] PATH_INFO: `/v3/users` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:44:19.649215 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffc57da7-a727-4066-9b57-9ca2b6a7aaa9 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81578) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:44:19.649570 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffc57da7-a727-4066-9b57-9ca2b6a7aaa9 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:44:19.650956 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffc57da7-a727-4066-9b57-9ca2b6a7aaa9 None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:44:19.668879 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 515/1040] 10.222.0.27 () {56 vars in 1133 bytes} [Tue Aug 30 01:44:19 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 30 01:44:19.674475 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffc57da7-a727-4066-9b57-9ca2b6a7aaa9 None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:44:19.693543 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:44:19.696991 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 526/1041] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Aug 30 01:44:19 2022] POST /identity/v3/auth/tokens => generated 1969 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Aug 30 01:44:19.791278 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffc57da7-a727-4066-9b57-9ca2b6a7aaa9 None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:44:19.825345 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35d69d97-2ce2-44e8-81a1-5dada95be759 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:44:19.825991 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35d69d97-2ce2-44e8-81a1-5dada95be759 None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:44:19.826272 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35d69d97-2ce2-44e8-81a1-5dada95be759 None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:44:19.826540 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35d69d97-2ce2-44e8-81a1-5dada95be759 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:44:19.896673 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35d69d97-2ce2-44e8-81a1-5dada95be759 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:44:19.898147 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35d69d97-2ce2-44e8-81a1-5dada95be759 None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:44:19.911489 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:44:19.965751 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88b42a3a-86b7-4470-b38c-e825f0e95c47 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:44:20.009649 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 516/1042] 10.222.0.27 () {58 vars in 1282 bytes} [Tue Aug 30 01:44:19 2022] GET /identity/v3/auth/tokens => generated 1969 bytes in 336 msecs (HTTP/1.1 200) 6 headers in 359 bytes (2 switches on core 0) Aug 30 01:44:20.012360 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35d69d97-2ce2-44e8-81a1-5dada95be759 None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:44:20.075063 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88b42a3a-86b7-4470-b38c-e825f0e95c47 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:44:20.100732 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:44:20.113272 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09654381-8f87-45bd-9ca8-e353b41d5e4e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:44:20.113911 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09654381-8f87-45bd-9ca8-e353b41d5e4e None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:44:20.114185 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09654381-8f87-45bd-9ca8-e353b41d5e4e None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:44:20.117627 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09654381-8f87-45bd-9ca8-e353b41d5e4e None admin] PATH_INFO: `/v3/services/image` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:44:20.117627 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09654381-8f87-45bd-9ca8-e353b41d5e4e None admin] RBAC: Authorizing `identity:get_service(service_id=image)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:44:20.117627 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09654381-8f87-45bd-9ca8-e353b41d5e4e None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:44:20.119004 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35d69d97-2ce2-44e8-81a1-5dada95be759 None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:44:20.125253 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application [None req-09654381-8f87-45bd-9ca8-e353b41d5e4e None admin] Could not find service: image.: keystone.exception.ServiceNotFound: Could not find service: image. Aug 30 01:44:20.125253 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 01:44:20.125253 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Aug 30 01:44:20.125253 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Aug 30 01:44:20.125253 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Aug 30 01:44:20.125253 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Aug 30 01:44:20.125253 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Aug 30 01:44:20.125253 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 30 01:44:20.125253 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Aug 30 01:44:20.125253 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Aug 30 01:44:20.125253 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 30 01:44:20.125253 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Aug 30 01:44:20.125253 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 01:44:20.125253 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 01:44:20.125253 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 01:44:20.125253 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 01:44:20.125253 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 01:44:20.126301 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 01:44:20.126301 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 01:44:20.126301 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 01:44:20.126301 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 01:44:20.126301 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 01:44:20.126301 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 01:44:20.126301 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Aug 30 01:44:20.126301 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_service(service_id) Aug 30 01:44:20.126301 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Aug 30 01:44:20.126301 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Aug 30 01:44:20.126301 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 01:44:20.126301 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 01:44:20.126301 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 01:44:20.126301 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 01:44:20.126301 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 01:44:20.126301 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 01:44:20.127182 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 01:44:20.127182 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application with Lock( Aug 30 01:44:20.127182 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 01:44:20.127182 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._enter() Aug 30 01:44:20.127182 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 01:44:20.127182 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 01:44:20.127182 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 01:44:20.127182 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.creator() Aug 30 01:44:20.127182 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 01:44:20.127182 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application created_value = creator( Aug 30 01:44:20.127182 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Aug 30 01:44:20.127182 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 30 01:44:20.127182 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Aug 30 01:44:20.127182 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Aug 30 01:44:20.129779 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 527/1043] 10.222.0.27 () {56 vars in 1105 bytes} [Tue Aug 30 01:44:19 2022] GET /identity/v3/services/image => generated 86 bytes in 164 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 01:44:20.132717 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09654381-8f87-45bd-9ca8-e353b41d5e4e None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:44:20.135352 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:44:20.159422 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8987eff6-4687-4268-984a-ef5d01e7b45f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:44:20.160111 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8987eff6-4687-4268-984a-ef5d01e7b45f None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:44:20.160396 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8987eff6-4687-4268-984a-ef5d01e7b45f None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:44:20.160663 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8987eff6-4687-4268-984a-ef5d01e7b45f None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:44:20.171652 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8987eff6-4687-4268-984a-ef5d01e7b45f None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:44:20.173402 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8987eff6-4687-4268-984a-ef5d01e7b45f None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:44:20.188997 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application [None req-8987eff6-4687-4268-984a-ef5d01e7b45f None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 01:44:20.188997 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 01:44:20.188997 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 01:44:20.188997 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 01:44:20.188997 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 01:44:20.188997 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 01:44:20.188997 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 01:44:20.188997 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 01:44:20.188997 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 01:44:20.188997 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 01:44:20.188997 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 01:44:20.188997 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 01:44:20.188997 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 30 01:44:20.188997 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 30 01:44:20.188997 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 30 01:44:20.188997 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 30 01:44:20.190016 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 01:44:20.190016 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 01:44:20.190016 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 01:44:20.190016 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 01:44:20.190016 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 01:44:20.190016 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 01:44:20.190016 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 01:44:20.190016 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application with Lock( Aug 30 01:44:20.190016 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 01:44:20.190016 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._enter() Aug 30 01:44:20.190016 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 01:44:20.190016 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 01:44:20.190016 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 01:44:20.190016 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.creator() Aug 30 01:44:20.190016 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 01:44:20.190016 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application created_value = creator( Aug 30 01:44:20.190016 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 30 01:44:20.190900 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 30 01:44:20.190900 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 01:44:20.190900 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application Aug 30 01:44:20.190900 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 517/1044] 10.222.0.27 () {56 vars in 1108 bytes} [Tue Aug 30 01:44:20 2022] GET /identity/v3/domains/Default => generated 87 bytes in 178 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 30 01:44:20.196205 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8987eff6-4687-4268-984a-ef5d01e7b45f None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:44:20.242329 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09654381-8f87-45bd-9ca8-e353b41d5e4e None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:44:20.278512 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6785ee30-cf66-4069-9e5e-356e606e6674 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:44:20.279133 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6785ee30-cf66-4069-9e5e-356e606e6674 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:44:20.279399 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6785ee30-cf66-4069-9e5e-356e606e6674 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:44:20.279670 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6785ee30-cf66-4069-9e5e-356e606e6674 None admin] PATH_INFO: `/v3/services` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:44:20.280300 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6785ee30-cf66-4069-9e5e-356e606e6674 None admin] RBAC: Adding query filter params (name=image) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:44:20.280605 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6785ee30-cf66-4069-9e5e-356e606e6674 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:44:20.282014 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6785ee30-cf66-4069-9e5e-356e606e6674 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:44:20.287569 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 528/1045] 10.222.0.27 () {56 vars in 1108 bytes} [Tue Aug 30 01:44:20 2022] GET /identity/v3/services?name=image => generated 122 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 30 01:44:20.293104 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6785ee30-cf66-4069-9e5e-356e606e6674 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:44:20.308470 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8987eff6-4687-4268-984a-ef5d01e7b45f None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:44:20.343023 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57b3c610-7079-4195-9d70-8f8627ccd7a8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:44:20.343636 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57b3c610-7079-4195-9d70-8f8627ccd7a8 None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:44:20.343911 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57b3c610-7079-4195-9d70-8f8627ccd7a8 None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:44:20.344181 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57b3c610-7079-4195-9d70-8f8627ccd7a8 None admin] PATH_INFO: `/v3/domains` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:44:20.344824 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57b3c610-7079-4195-9d70-8f8627ccd7a8 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81578) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:44:20.345126 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57b3c610-7079-4195-9d70-8f8627ccd7a8 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:44:20.346513 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57b3c610-7079-4195-9d70-8f8627ccd7a8 None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:44:20.352291 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:44:20.364195 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 518/1046] 10.222.0.27 () {56 vars in 1109 bytes} [Tue Aug 30 01:44:20 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 30 01:44:20.365779 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57b3c610-7079-4195-9d70-8f8627ccd7a8 None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:44:20.392176 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:44:20.414334 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6785ee30-cf66-4069-9e5e-356e606e6674 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:44:20.452359 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-deb82515-56b1-4395-9ffe-ccb2e481a77b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:44:20.452972 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-deb82515-56b1-4395-9ffe-ccb2e481a77b None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:44:20.453260 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-deb82515-56b1-4395-9ffe-ccb2e481a77b None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:44:20.453557 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-deb82515-56b1-4395-9ffe-ccb2e481a77b None admin] PATH_INFO: `/v3/services` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:44:20.454195 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-deb82515-56b1-4395-9ffe-ccb2e481a77b None admin] RBAC: Adding query filter params (type=image) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:44:20.454503 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-deb82515-56b1-4395-9ffe-ccb2e481a77b None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:44:20.455881 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-deb82515-56b1-4395-9ffe-ccb2e481a77b None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:44:20.461351 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 529/1047] 10.222.0.27 () {56 vars in 1108 bytes} [Tue Aug 30 01:44:20 2022] GET /identity/v3/services?type=image => generated 350 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 30 01:44:20.467192 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-deb82515-56b1-4395-9ffe-ccb2e481a77b None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:44:20.481545 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57b3c610-7079-4195-9d70-8f8627ccd7a8 None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:44:20.518213 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d91c1a9-9534-4615-877f-b5bdd95a743a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:44:20.519680 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d91c1a9-9534-4615-877f-b5bdd95a743a None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:44:20.520031 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d91c1a9-9534-4615-877f-b5bdd95a743a None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:44:20.520303 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d91c1a9-9534-4615-877f-b5bdd95a743a None admin] PATH_INFO: `/v3/projects/service` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:44:20.526974 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d91c1a9-9534-4615-877f-b5bdd95a743a None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:44:20.528381 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d91c1a9-9534-4615-877f-b5bdd95a743a None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:44:20.539846 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application [None req-3d91c1a9-9534-4615-877f-b5bdd95a743a None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 30 01:44:20.539846 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 01:44:20.539846 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 01:44:20.539846 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 01:44:20.539846 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 01:44:20.539846 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 01:44:20.539846 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 01:44:20.539846 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 01:44:20.539846 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 01:44:20.539846 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 01:44:20.539846 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 01:44:20.539846 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 01:44:20.539846 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 30 01:44:20.539846 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 30 01:44:20.539846 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 30 01:44:20.540744 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 30 01:44:20.540744 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 01:44:20.540744 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 01:44:20.540744 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 01:44:20.540744 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 01:44:20.540744 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 01:44:20.540744 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 01:44:20.540744 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 01:44:20.540744 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application with Lock( Aug 30 01:44:20.540744 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 01:44:20.540744 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._enter() Aug 30 01:44:20.540744 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 01:44:20.540744 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 01:44:20.540744 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 01:44:20.540744 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.creator() Aug 30 01:44:20.540744 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 01:44:20.540744 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application created_value = creator( Aug 30 01:44:20.541703 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 30 01:44:20.541703 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 30 01:44:20.541703 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 30 01:44:20.541703 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 30 01:44:20.541703 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 30 01:44:20.541703 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 30 01:44:20.541703 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 30 01:44:20.541703 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application Aug 30 01:44:20.541703 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 519/1048] 10.222.0.27 () {56 vars in 1111 bytes} [Tue Aug 30 01:44:20 2022] GET /identity/v3/projects/service => generated 88 bytes in 176 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 30 01:44:20.546855 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d91c1a9-9534-4615-877f-b5bdd95a743a None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:44:20.577995 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-deb82515-56b1-4395-9ffe-ccb2e481a77b None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:44:20.595291 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:44:20.616369 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1127d33a-8c6d-43b5-a147-6de362d9e83d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:44:20.616978 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1127d33a-8c6d-43b5-a147-6de362d9e83d None admin] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:44:20.617258 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1127d33a-8c6d-43b5-a147-6de362d9e83d None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:44:20.618291 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1127d33a-8c6d-43b5-a147-6de362d9e83d None admin] PATH_INFO: `/v3/endpoints` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:44:20.618291 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1127d33a-8c6d-43b5-a147-6de362d9e83d None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:44:20.619650 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1127d33a-8c6d-43b5-a147-6de362d9e83d None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:44:20.632761 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:44:20.648595 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d91c1a9-9534-4615-877f-b5bdd95a743a None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:44:20.682545 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1e393fa-e1b1-4a69-bcbf-cf7ba3b62e68 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:44:20.683128 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1e393fa-e1b1-4a69-bcbf-cf7ba3b62e68 None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:44:20.683484 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1e393fa-e1b1-4a69-bcbf-cf7ba3b62e68 None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:44:20.683757 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1e393fa-e1b1-4a69-bcbf-cf7ba3b62e68 None admin] PATH_INFO: `/v3/projects` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:44:20.684449 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1e393fa-e1b1-4a69-bcbf-cf7ba3b62e68 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81578) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:44:20.684756 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1e393fa-e1b1-4a69-bcbf-cf7ba3b62e68 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:44:20.686200 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1e393fa-e1b1-4a69-bcbf-cf7ba3b62e68 None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:44:20.687285 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 530/1049] 10.222.0.27 () {60 vars in 1144 bytes} [Tue Aug 30 01:44:20 2022] POST /identity/v3/endpoints => generated 327 bytes in 220 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 30 01:44:20.700979 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 520/1050] 10.222.0.27 () {56 vars in 1148 bytes} [Tue Aug 30 01:44:20 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:44:20.709909 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1127d33a-8c6d-43b5-a147-6de362d9e83d None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:44:20.830118 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1127d33a-8c6d-43b5-a147-6de362d9e83d None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:44:20.868525 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8c10a15-9274-4571-a3f3-62717cfc916e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:44:20.869131 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8c10a15-9274-4571-a3f3-62717cfc916e None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:44:20.869409 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8c10a15-9274-4571-a3f3-62717cfc916e None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:44:20.869703 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8c10a15-9274-4571-a3f3-62717cfc916e None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:44:20.870430 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8c10a15-9274-4571-a3f3-62717cfc916e None admin] RBAC: Adding query filter params (role.id=8e1192374afc4af6a7cea01c69e72402, scope.project.id=4562f3951fbe465280073e60a11e44e2, user.id=d64f2e71361644589bec8a06e6eceed1) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:44:20.870737 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8c10a15-9274-4571-a3f3-62717cfc916e None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:44:20.872155 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8c10a15-9274-4571-a3f3-62717cfc916e None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:44:20.878279 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 531/1051] 10.222.0.27 () {56 vars in 1374 bytes} [Tue Aug 30 01:44:20 2022] GET /identity/v3/role_assignments?user.id=d64f2e71361644589bec8a06e6eceed1&scope.project.id=4562f3951fbe465280073e60a11e44e2&role.id=8e1192374afc4af6a7cea01c69e72402 => generated 259 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:44:21.299088 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e536b42-1caa-40c8-90a7-996ed72e65ac None None] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:44:21.299418 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e536b42-1caa-40c8-90a7-996ed72e65ac None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:44:21.299654 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e536b42-1caa-40c8-90a7-996ed72e65ac None None] PATH_INFO: `/` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:44:21.301706 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 521/1052] 10.222.0.27 () {54 vars in 927 bytes} [Tue Aug 30 01:44:21 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 01:44:21.307156 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4314a212-9178-4211-ada5-475e92d85fbd None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:44:21.307617 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4314a212-9178-4211-ada5-475e92d85fbd None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:44:21.307708 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4314a212-9178-4211-ada5-475e92d85fbd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:44:21.328582 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-4314a212-9178-4211-ada5-475e92d85fbd None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 01:44:21.329810 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-4314a212-9178-4211-ada5-475e92d85fbd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:44:21.356275 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:44:21.404769 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 532/1053] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Aug 30 01:44:21 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 30 01:44:21.685209 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e536b42-1caa-40c8-90a7-996ed72e65ac None None] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:44:21.798130 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e536b42-1caa-40c8-90a7-996ed72e65ac None None] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:44:21.834408 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f337f54-390f-4221-a04c-c1828f292b82 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:44:21.835041 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f337f54-390f-4221-a04c-c1828f292b82 None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:44:21.835317 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f337f54-390f-4221-a04c-c1828f292b82 None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:44:21.835586 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f337f54-390f-4221-a04c-c1828f292b82 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:44:21.846252 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f337f54-390f-4221-a04c-c1828f292b82 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 30 01:44:21.852012 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application [None req-8f337f54-390f-4221-a04c-c1828f292b82 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 30 01:44:21.852012 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 01:44:21.852012 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 01:44:21.852012 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 01:44:21.852012 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 01:44:21.852012 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 01:44:21.852012 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 01:44:21.852012 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 01:44:21.852012 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 01:44:21.852012 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 01:44:21.852012 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 01:44:21.852012 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 01:44:21.852012 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 30 01:44:21.852012 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 30 01:44:21.852012 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 30 01:44:21.852012 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 30 01:44:21.852954 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 30 01:44:21.852954 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 30 01:44:21.852954 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 30 01:44:21.852954 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 30 01:44:21.852954 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 01:44:21.852954 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 01:44:21.852954 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 01:44:21.852954 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 01:44:21.852954 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 01:44:21.852954 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 01:44:21.852954 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 01:44:21.852954 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application with Lock( Aug 30 01:44:21.852954 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 01:44:21.852954 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._enter() Aug 30 01:44:21.852954 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 01:44:21.852954 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 01:44:21.853979 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 01:44:21.853979 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.creator() Aug 30 01:44:21.853979 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 01:44:21.853979 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application created_value = creator( Aug 30 01:44:21.853979 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 30 01:44:21.853979 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 30 01:44:21.853979 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 30 01:44:21.853979 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 30 01:44:21.853979 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 30 01:44:21.853979 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 30 01:44:21.853979 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 30 01:44:21.853979 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application Aug 30 01:44:21.853979 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 522/1054] 10.222.0.27 () {56 vars in 1096 bytes} [Tue Aug 30 01:44:21 2022] GET /identity/v3/roles/admin => generated 83 bytes in 169 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 01:44:21.859031 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4314a212-9178-4211-ada5-475e92d85fbd None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:44:21.965603 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4314a212-9178-4211-ada5-475e92d85fbd None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:44:22.000774 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-189d4be1-3001-49cc-97d8-5a40cc86bcb0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:44:22.001380 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-189d4be1-3001-49cc-97d8-5a40cc86bcb0 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:44:22.001683 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-189d4be1-3001-49cc-97d8-5a40cc86bcb0 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:44:22.001955 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-189d4be1-3001-49cc-97d8-5a40cc86bcb0 None admin] PATH_INFO: `/v3/roles` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:44:22.002616 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-189d4be1-3001-49cc-97d8-5a40cc86bcb0 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:44:22.002916 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-189d4be1-3001-49cc-97d8-5a40cc86bcb0 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:44:22.004916 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-189d4be1-3001-49cc-97d8-5a40cc86bcb0 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:44:22.013329 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 533/1055] 10.222.0.27 () {56 vars in 1099 bytes} [Tue Aug 30 01:44:21 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 30 01:44:22.018828 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f337f54-390f-4221-a04c-c1828f292b82 None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:44:22.023363 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:44:22.120742 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f337f54-390f-4221-a04c-c1828f292b82 None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:44:22.154283 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5ab013c-b6c7-4c58-8665-dc706f1e3ddc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:44:22.154907 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5ab013c-b6c7-4c58-8665-dc706f1e3ddc None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:44:22.155185 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5ab013c-b6c7-4c58-8665-dc706f1e3ddc None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:44:22.155456 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5ab013c-b6c7-4c58-8665-dc706f1e3ddc None admin] PATH_INFO: `/v3/users/7ce2deb93a6249aa96a701b6bd690844` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:44:22.169136 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5ab013c-b6c7-4c58-8665-dc706f1e3ddc None admin] RBAC: Authorizing `identity:get_user(user_id=7ce2deb93a6249aa96a701b6bd690844)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:44:22.170608 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5ab013c-b6c7-4c58-8665-dc706f1e3ddc None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:44:22.172391 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 523/1056] 10.222.0.27 () {56 vars in 1177 bytes} [Tue Aug 30 01:44:22 2022] GET /identity/v3/users/7ce2deb93a6249aa96a701b6bd690844 => generated 285 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:44:22.178248 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-189d4be1-3001-49cc-97d8-5a40cc86bcb0 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:44:22.202995 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:44:22.279563 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-189d4be1-3001-49cc-97d8-5a40cc86bcb0 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:44:22.313177 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79d2a824-0454-4afd-b0ef-ecd1721e2b19 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:44:22.313892 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79d2a824-0454-4afd-b0ef-ecd1721e2b19 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:44:22.314104 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79d2a824-0454-4afd-b0ef-ecd1721e2b19 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:44:22.314342 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79d2a824-0454-4afd-b0ef-ecd1721e2b19 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:44:22.384640 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79d2a824-0454-4afd-b0ef-ecd1721e2b19 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:44:22.386122 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79d2a824-0454-4afd-b0ef-ecd1721e2b19 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:44:22.409167 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:44:22.492074 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 534/1057] 10.222.0.27 () {58 vars in 1282 bytes} [Tue Aug 30 01:44:22 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 309 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Aug 30 01:44:22.493382 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5ab013c-b6c7-4c58-8665-dc706f1e3ddc None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:44:22.556760 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:44:22.594688 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5ab013c-b6c7-4c58-8665-dc706f1e3ddc None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:44:22.629948 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53e9e6cf-9bf0-497b-9ecd-8096d8a38359 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:44:22.630575 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53e9e6cf-9bf0-497b-9ecd-8096d8a38359 None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:44:22.630849 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53e9e6cf-9bf0-497b-9ecd-8096d8a38359 None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:44:22.631112 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53e9e6cf-9bf0-497b-9ecd-8096d8a38359 None admin] PATH_INFO: `/v3/projects/c09a62e9f672418589e36f7395117fdb` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:44:22.632208 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53e9e6cf-9bf0-497b-9ecd-8096d8a38359 None admin] RBAC: Authorizing `identity:get_project(project_id=c09a62e9f672418589e36f7395117fdb)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:44:22.633650 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53e9e6cf-9bf0-497b-9ecd-8096d8a38359 None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:44:22.635465 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 524/1058] 10.222.0.27 () {56 vars in 1186 bytes} [Tue Aug 30 01:44:22 2022] GET /identity/v3/projects/c09a62e9f672418589e36f7395117fdb => generated 311 bytes in 142 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:44:22.641313 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79d2a824-0454-4afd-b0ef-ecd1721e2b19 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:44:22.686820 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c103729-cd81-4fab-9c6d-b5e3a2aaf2f7 None None] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:44:22.687348 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c103729-cd81-4fab-9c6d-b5e3a2aaf2f7 None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:44:22.687348 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c103729-cd81-4fab-9c6d-b5e3a2aaf2f7 None None] PATH_INFO: `/` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:44:22.689159 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 525/1059] 10.222.0.27 () {54 vars in 927 bytes} [Tue Aug 30 01:44:22 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 30 01:44:22.694152 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0176e31-c1f2-4bb6-be8f-12a26c975005 None None] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:44:22.694440 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0176e31-c1f2-4bb6-be8f-12a26c975005 None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:44:22.694716 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0176e31-c1f2-4bb6-be8f-12a26c975005 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:44:22.724129 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.auth.core [None req-e0176e31-c1f2-4bb6-be8f-12a26c975005 None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81578) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 01:44:22.726804 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-e0176e31-c1f2-4bb6-be8f-12a26c975005 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:44:22.761323 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79d2a824-0454-4afd-b0ef-ecd1721e2b19 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:44:22.783964 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:44:22.796020 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f1447c8-ca26-42b6-ae01-608ef8c5afc5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:44:22.797433 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f1447c8-ca26-42b6-ae01-608ef8c5afc5 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:44:22.797433 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f1447c8-ca26-42b6-ae01-608ef8c5afc5 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:44:22.797433 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f1447c8-ca26-42b6-ae01-608ef8c5afc5 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:44:22.797890 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f1447c8-ca26-42b6-ae01-608ef8c5afc5 None admin] RBAC: Adding query filter params (role.id=ead5fac86a0747ea8823a82efcea8ed7, scope.project.id=c09a62e9f672418589e36f7395117fdb, user.id=7ce2deb93a6249aa96a701b6bd690844) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:44:22.798198 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f1447c8-ca26-42b6-ae01-608ef8c5afc5 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:44:22.799621 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f1447c8-ca26-42b6-ae01-608ef8c5afc5 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:44:22.802969 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 526/1060] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Aug 30 01:44:22 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Aug 30 01:44:22.808068 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 535/1061] 10.222.0.27 () {56 vars in 1374 bytes} [Tue Aug 30 01:44:22 2022] GET /identity/v3/role_assignments?user.id=7ce2deb93a6249aa96a701b6bd690844&scope.project.id=c09a62e9f672418589e36f7395117fdb&role.id=ead5fac86a0747ea8823a82efcea8ed7 => generated 259 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:44:22.825898 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84c41a64-de8d-42eb-9daa-58643b6be83c None None] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:44:22.826187 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84c41a64-de8d-42eb-9daa-58643b6be83c None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:44:22.826628 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84c41a64-de8d-42eb-9daa-58643b6be83c None None] PATH_INFO: `/` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:44:22.828110 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 527/1062] 10.222.0.27 () {54 vars in 927 bytes} [Tue Aug 30 01:44:22 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 01:44:22.834923 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dc4fa1b-b240-4d00-91bd-ef1e4b743b36 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:44:22.835236 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dc4fa1b-b240-4d00-91bd-ef1e4b743b36 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:44:22.835510 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dc4fa1b-b240-4d00-91bd-ef1e4b743b36 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:44:22.856105 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-4dc4fa1b-b240-4d00-91bd-ef1e4b743b36 None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 01:44:22.857345 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-4dc4fa1b-b240-4d00-91bd-ef1e4b743b36 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:44:22.932681 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 536/1063] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Aug 30 01:44:22 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 30 01:44:23.079427 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84c41a64-de8d-42eb-9daa-58643b6be83c None None] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:44:23.143294 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:44:23.184009 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84c41a64-de8d-42eb-9daa-58643b6be83c None None] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:44:23.209819 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4dc4fa1b-b240-4d00-91bd-ef1e4b743b36 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:44:23.219389 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb4fc015-2e22-400f-9d57-47719ade259c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:44:23.220108 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb4fc015-2e22-400f-9d57-47719ade259c None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:44:23.220383 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb4fc015-2e22-400f-9d57-47719ade259c None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:44:23.220650 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb4fc015-2e22-400f-9d57-47719ade259c None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:44:23.298004 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb4fc015-2e22-400f-9d57-47719ade259c None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:44:23.299853 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb4fc015-2e22-400f-9d57-47719ade259c None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:44:23.317486 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4dc4fa1b-b240-4d00-91bd-ef1e4b743b36 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:44:23.338794 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:44:23.357909 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bee0bb0-4378-4514-9606-1d5717ed3432 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:44:23.358841 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bee0bb0-4378-4514-9606-1d5717ed3432 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:44:23.359124 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bee0bb0-4378-4514-9606-1d5717ed3432 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:44:23.359395 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bee0bb0-4378-4514-9606-1d5717ed3432 None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:44:23.363421 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:44:23.369901 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bee0bb0-4378-4514-9606-1d5717ed3432 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 30 01:44:23.376852 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application [None req-6bee0bb0-4378-4514-9606-1d5717ed3432 None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Aug 30 01:44:23.376852 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 01:44:23.376852 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 01:44:23.376852 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 01:44:23.376852 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 01:44:23.376852 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 01:44:23.376852 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 01:44:23.376852 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 01:44:23.376852 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 01:44:23.376852 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 01:44:23.376852 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 01:44:23.376852 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 01:44:23.376852 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 30 01:44:23.376852 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 30 01:44:23.376852 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 30 01:44:23.376852 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 30 01:44:23.378005 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 30 01:44:23.378005 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 30 01:44:23.378005 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 30 01:44:23.378005 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 30 01:44:23.378005 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 01:44:23.378005 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 01:44:23.378005 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 01:44:23.378005 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 01:44:23.378005 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 01:44:23.378005 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 01:44:23.378005 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 01:44:23.378005 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application with Lock( Aug 30 01:44:23.378005 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 01:44:23.378005 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._enter() Aug 30 01:44:23.378005 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 01:44:23.378005 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 01:44:23.379920 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 01:44:23.379920 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.creator() Aug 30 01:44:23.379920 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 01:44:23.379920 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application created_value = creator( Aug 30 01:44:23.379920 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 30 01:44:23.379920 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 30 01:44:23.379920 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 30 01:44:23.379920 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 30 01:44:23.379920 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 30 01:44:23.379920 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 30 01:44:23.379920 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Aug 30 01:44:23.379920 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Aug 30 01:44:23.385769 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 537/1064] 10.222.0.27 () {56 vars in 1120 bytes} [Tue Aug 30 01:44:23 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 176 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 30 01:44:23.391115 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bee0bb0-4378-4514-9606-1d5717ed3432 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:44:23.411345 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 528/1065] 10.222.0.27 () {58 vars in 1282 bytes} [Tue Aug 30 01:44:23 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 332 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Aug 30 01:44:23.414095 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb4fc015-2e22-400f-9d57-47719ade259c None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:44:23.504512 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bee0bb0-4378-4514-9606-1d5717ed3432 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:44:23.521138 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb4fc015-2e22-400f-9d57-47719ade259c None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:44:23.539700 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbc3efa8-28a4-417e-a9f8-7ec71abd70b9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:44:23.540299 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbc3efa8-28a4-417e-a9f8-7ec71abd70b9 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:44:23.540575 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbc3efa8-28a4-417e-a9f8-7ec71abd70b9 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:44:23.540839 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbc3efa8-28a4-417e-a9f8-7ec71abd70b9 None admin] PATH_INFO: `/v3/roles` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:44:23.541519 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbc3efa8-28a4-417e-a9f8-7ec71abd70b9 None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:44:23.541822 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbc3efa8-28a4-417e-a9f8-7ec71abd70b9 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:44:23.543174 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbc3efa8-28a4-417e-a9f8-7ec71abd70b9 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:44:23.551997 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 538/1066] 10.222.0.27 () {56 vars in 1115 bytes} [Tue Aug 30 01:44:23 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 30 01:44:23.557845 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbc3efa8-28a4-417e-a9f8-7ec71abd70b9 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:44:23.559677 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc64513b-02aa-4570-acf0-0775aa467871 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:44:23.560376 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc64513b-02aa-4570-acf0-0775aa467871 None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:44:23.560653 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc64513b-02aa-4570-acf0-0775aa467871 None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:44:23.560921 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc64513b-02aa-4570-acf0-0775aa467871 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:44:23.571823 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc64513b-02aa-4570-acf0-0775aa467871 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:44:23.573242 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc64513b-02aa-4570-acf0-0775aa467871 None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:44:23.587815 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application [None req-dc64513b-02aa-4570-acf0-0775aa467871 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 01:44:23.587815 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 01:44:23.587815 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 01:44:23.587815 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 01:44:23.587815 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 01:44:23.587815 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 01:44:23.587815 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 01:44:23.587815 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 01:44:23.587815 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 01:44:23.587815 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 01:44:23.587815 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 01:44:23.587815 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 01:44:23.587815 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 30 01:44:23.587815 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 30 01:44:23.587815 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 30 01:44:23.587815 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 30 01:44:23.588940 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 01:44:23.588940 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 01:44:23.588940 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 01:44:23.588940 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 01:44:23.588940 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 01:44:23.588940 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 01:44:23.588940 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 01:44:23.588940 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application with Lock( Aug 30 01:44:23.588940 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 01:44:23.588940 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._enter() Aug 30 01:44:23.588940 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 01:44:23.588940 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 01:44:23.588940 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 01:44:23.588940 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.creator() Aug 30 01:44:23.588940 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 01:44:23.588940 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application created_value = creator( Aug 30 01:44:23.588940 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 30 01:44:23.590651 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 30 01:44:23.590651 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 01:44:23.590651 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application Aug 30 01:44:23.590651 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 529/1067] 10.222.0.27 () {56 vars in 1108 bytes} [Tue Aug 30 01:44:23 2022] GET /identity/v3/domains/Default => generated 87 bytes in 175 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 01:44:23.594795 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc64513b-02aa-4570-acf0-0775aa467871 None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:44:23.602908 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:44:23.610673 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:44:23.665023 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbc3efa8-28a4-417e-a9f8-7ec71abd70b9 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:44:23.704814 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65075f4c-1213-400d-92c3-43c79cdb94f0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:44:23.705050 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc64513b-02aa-4570-acf0-0775aa467871 None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:44:23.705480 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65075f4c-1213-400d-92c3-43c79cdb94f0 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:44:23.705480 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65075f4c-1213-400d-92c3-43c79cdb94f0 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:44:23.705480 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65075f4c-1213-400d-92c3-43c79cdb94f0 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:44:23.717846 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65075f4c-1213-400d-92c3-43c79cdb94f0 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:44:23.719439 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65075f4c-1213-400d-92c3-43c79cdb94f0 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:44:23.735630 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application [None req-65075f4c-1213-400d-92c3-43c79cdb94f0 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 01:44:23.735630 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 01:44:23.735630 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 01:44:23.735630 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 01:44:23.735630 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 01:44:23.735630 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 01:44:23.735630 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 01:44:23.735630 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 01:44:23.735630 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 01:44:23.735630 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 01:44:23.735630 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 01:44:23.735630 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 01:44:23.735630 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 30 01:44:23.735630 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 30 01:44:23.735630 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 30 01:44:23.735630 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 30 01:44:23.737604 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 01:44:23.737604 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 01:44:23.737604 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 01:44:23.737604 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 01:44:23.737604 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 01:44:23.737604 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 01:44:23.737604 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 01:44:23.737604 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application with Lock( Aug 30 01:44:23.737604 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 01:44:23.737604 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._enter() Aug 30 01:44:23.737604 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 01:44:23.737604 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 01:44:23.737604 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 01:44:23.737604 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.creator() Aug 30 01:44:23.737604 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 01:44:23.737604 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application created_value = creator( Aug 30 01:44:23.737604 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 30 01:44:23.739529 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 30 01:44:23.739529 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 01:44:23.739529 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Aug 30 01:44:23.739529 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 539/1068] 10.222.0.27 () {56 vars in 1108 bytes} [Tue Aug 30 01:44:23 2022] GET /identity/v3/domains/Default => generated 87 bytes in 180 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 30 01:44:23.743746 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65075f4c-1213-400d-92c3-43c79cdb94f0 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:44:23.754189 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44baec3d-6501-4595-b8e1-0be3cc75b6b1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:44:23.754980 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44baec3d-6501-4595-b8e1-0be3cc75b6b1 None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:44:23.755264 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44baec3d-6501-4595-b8e1-0be3cc75b6b1 None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:44:23.755543 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44baec3d-6501-4595-b8e1-0be3cc75b6b1 None admin] PATH_INFO: `/v3/domains` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:44:23.756196 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44baec3d-6501-4595-b8e1-0be3cc75b6b1 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81578) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:44:23.756505 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44baec3d-6501-4595-b8e1-0be3cc75b6b1 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:44:23.757993 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44baec3d-6501-4595-b8e1-0be3cc75b6b1 None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:44:23.773504 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 530/1069] 10.222.0.27 () {56 vars in 1109 bytes} [Tue Aug 30 01:44:23 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 30 01:44:23.846215 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:44:23.857195 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65075f4c-1213-400d-92c3-43c79cdb94f0 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:44:23.898253 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad2121c8-af7e-48b6-ac76-ee1744eb7a6a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:44:23.898897 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad2121c8-af7e-48b6-ac76-ee1744eb7a6a None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:44:23.899179 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad2121c8-af7e-48b6-ac76-ee1744eb7a6a None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:44:23.899452 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad2121c8-af7e-48b6-ac76-ee1744eb7a6a None admin] PATH_INFO: `/v3/domains` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:44:23.900138 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad2121c8-af7e-48b6-ac76-ee1744eb7a6a None admin] RBAC: Adding query filter params (name=Default) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:44:23.900446 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad2121c8-af7e-48b6-ac76-ee1744eb7a6a None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:44:23.901957 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad2121c8-af7e-48b6-ac76-ee1744eb7a6a None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:44:23.917800 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 540/1070] 10.222.0.27 () {56 vars in 1109 bytes} [Tue Aug 30 01:44:23 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 30 01:44:23.923173 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44baec3d-6501-4595-b8e1-0be3cc75b6b1 None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:44:24.002376 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:44:24.034501 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44baec3d-6501-4595-b8e1-0be3cc75b6b1 None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:44:24.073513 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b587173-3745-4550-8d51-59629ca5c1e3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:44:24.074151 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b587173-3745-4550-8d51-59629ca5c1e3 None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:44:24.074424 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b587173-3745-4550-8d51-59629ca5c1e3 None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:44:24.074691 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b587173-3745-4550-8d51-59629ca5c1e3 None admin] PATH_INFO: `/v3/users/nova` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:44:24.082359 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b587173-3745-4550-8d51-59629ca5c1e3 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:44:24.083956 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b587173-3745-4550-8d51-59629ca5c1e3 None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:44:24.095727 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application [None req-2b587173-3745-4550-8d51-59629ca5c1e3 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Aug 30 01:44:24.095727 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 01:44:24.095727 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 01:44:24.095727 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 01:44:24.095727 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 01:44:24.095727 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 01:44:24.095727 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 01:44:24.095727 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 01:44:24.095727 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 01:44:24.095727 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 01:44:24.095727 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 01:44:24.095727 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 01:44:24.095727 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 30 01:44:24.095727 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 30 01:44:24.095727 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 30 01:44:24.095727 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 30 01:44:24.096835 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 01:44:24.096835 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 01:44:24.096835 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 30 01:44:24.096835 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 30 01:44:24.096835 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 30 01:44:24.096835 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 30 01:44:24.096835 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 01:44:24.096835 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 01:44:24.096835 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 01:44:24.096835 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 01:44:24.096835 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 01:44:24.096835 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application with Lock( Aug 30 01:44:24.096835 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 01:44:24.096835 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._enter() Aug 30 01:44:24.096835 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 01:44:24.096835 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 01:44:24.096835 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 01:44:24.096835 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.creator() Aug 30 01:44:24.098860 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 01:44:24.098860 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application created_value = creator( Aug 30 01:44:24.098860 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 30 01:44:24.098860 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 30 01:44:24.098860 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 30 01:44:24.098860 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 30 01:44:24.098860 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 30 01:44:24.098860 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 30 01:44:24.098860 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Aug 30 01:44:24.098860 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application Aug 30 01:44:24.098860 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 531/1071] 10.222.0.27 () {56 vars in 1093 bytes} [Tue Aug 30 01:44:23 2022] GET /identity/v3/users/nova => generated 82 bytes in 174 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 30 01:44:24.103575 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad2121c8-af7e-48b6-ac76-ee1744eb7a6a None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:44:24.201805 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad2121c8-af7e-48b6-ac76-ee1744eb7a6a None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:44:24.235741 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a19daea1-88e1-45ae-9cf9-a0485fc569e0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:44:24.236418 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a19daea1-88e1-45ae-9cf9-a0485fc569e0 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:44:24.236699 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a19daea1-88e1-45ae-9cf9-a0485fc569e0 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:44:24.236967 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a19daea1-88e1-45ae-9cf9-a0485fc569e0 None admin] PATH_INFO: `/v3/users` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:44:24.238217 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a19daea1-88e1-45ae-9cf9-a0485fc569e0 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:44:24.238525 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a19daea1-88e1-45ae-9cf9-a0485fc569e0 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:44:24.240133 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a19daea1-88e1-45ae-9cf9-a0485fc569e0 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:44:24.257608 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 541/1072] 10.222.0.27 () {56 vars in 1133 bytes} [Tue Aug 30 01:44:24 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:44:24.263371 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b587173-3745-4550-8d51-59629ca5c1e3 None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:44:24.361374 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b587173-3745-4550-8d51-59629ca5c1e3 None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:44:24.390581 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:44:24.395413 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-550f8b10-61fd-49b5-a1cd-a8e250bbfc74 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:44:24.396033 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-550f8b10-61fd-49b5-a1cd-a8e250bbfc74 None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:44:24.396308 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-550f8b10-61fd-49b5-a1cd-a8e250bbfc74 None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:44:24.396575 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-550f8b10-61fd-49b5-a1cd-a8e250bbfc74 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:44:24.406850 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-550f8b10-61fd-49b5-a1cd-a8e250bbfc74 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:44:24.408264 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-550f8b10-61fd-49b5-a1cd-a8e250bbfc74 None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:44:24.422660 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application [None req-550f8b10-61fd-49b5-a1cd-a8e250bbfc74 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 01:44:24.422660 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 01:44:24.422660 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 01:44:24.422660 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 01:44:24.422660 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 01:44:24.422660 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 01:44:24.422660 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 01:44:24.422660 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 01:44:24.422660 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 01:44:24.422660 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 01:44:24.422660 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 01:44:24.422660 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 01:44:24.422660 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 30 01:44:24.422660 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 30 01:44:24.422660 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 30 01:44:24.422660 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 30 01:44:24.423640 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 01:44:24.423640 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 01:44:24.423640 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 01:44:24.423640 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 01:44:24.423640 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 01:44:24.423640 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 01:44:24.423640 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 01:44:24.423640 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application with Lock( Aug 30 01:44:24.423640 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 01:44:24.423640 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._enter() Aug 30 01:44:24.423640 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 01:44:24.423640 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 01:44:24.423640 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 01:44:24.423640 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.creator() Aug 30 01:44:24.423640 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 01:44:24.423640 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application created_value = creator( Aug 30 01:44:24.423640 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 30 01:44:24.424537 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 30 01:44:24.424537 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 01:44:24.424537 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application Aug 30 01:44:24.424537 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 532/1073] 10.222.0.27 () {56 vars in 1108 bytes} [Tue Aug 30 01:44:24 2022] GET /identity/v3/domains/Default => generated 87 bytes in 161 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 01:44:24.430343 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a19daea1-88e1-45ae-9cf9-a0485fc569e0 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:44:24.441597 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:44:24.531220 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a19daea1-88e1-45ae-9cf9-a0485fc569e0 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:44:24.564360 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6c0366b-59df-4866-b005-a6b7ab7bf969 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:44:24.564952 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6c0366b-59df-4866-b005-a6b7ab7bf969 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:44:24.565226 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6c0366b-59df-4866-b005-a6b7ab7bf969 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:44:24.565625 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6c0366b-59df-4866-b005-a6b7ab7bf969 None admin] PATH_INFO: `/v3/domains` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:44:24.566876 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6c0366b-59df-4866-b005-a6b7ab7bf969 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:44:24.567171 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6c0366b-59df-4866-b005-a6b7ab7bf969 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:44:24.568518 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6c0366b-59df-4866-b005-a6b7ab7bf969 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:44:24.579607 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 542/1074] 10.222.0.27 () {56 vars in 1109 bytes} [Tue Aug 30 01:44:24 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:44:24.586040 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-550f8b10-61fd-49b5-a1cd-a8e250bbfc74 None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:44:24.686712 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-550f8b10-61fd-49b5-a1cd-a8e250bbfc74 None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:44:24.721185 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4545918c-94b4-48a7-b73d-5b3a74007e3a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:44:24.721784 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4545918c-94b4-48a7-b73d-5b3a74007e3a None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:44:24.721784 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4545918c-94b4-48a7-b73d-5b3a74007e3a None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:44:24.722611 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4545918c-94b4-48a7-b73d-5b3a74007e3a None admin] PATH_INFO: `/v3/projects/service` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:44:24.729778 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4545918c-94b4-48a7-b73d-5b3a74007e3a None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:44:24.731213 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4545918c-94b4-48a7-b73d-5b3a74007e3a None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:44:24.734406 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29fc522e-0f8c-460f-afbe-eacc2698c8ea None None] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:44:24.734801 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29fc522e-0f8c-460f-afbe-eacc2698c8ea None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:44:24.735076 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29fc522e-0f8c-460f-afbe-eacc2698c8ea None None] PATH_INFO: `/` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:44:24.736416 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 543/1075] 10.222.0.27 () {54 vars in 927 bytes} [Tue Aug 30 01:44:24 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 30 01:44:24.742016 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18cfd0fc-e854-4e82-992a-da794f76b1df None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:44:24.742298 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18cfd0fc-e854-4e82-992a-da794f76b1df None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:44:24.742567 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18cfd0fc-e854-4e82-992a-da794f76b1df None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:44:24.743677 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application [None req-4545918c-94b4-48a7-b73d-5b3a74007e3a None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 30 01:44:24.743677 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 01:44:24.743677 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 01:44:24.743677 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 01:44:24.743677 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 01:44:24.743677 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 01:44:24.743677 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 01:44:24.743677 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 01:44:24.743677 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 01:44:24.743677 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 01:44:24.743677 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 01:44:24.743677 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 01:44:24.743677 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 30 01:44:24.743677 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 30 01:44:24.743677 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 30 01:44:24.744606 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 30 01:44:24.744606 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 01:44:24.744606 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 01:44:24.744606 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 01:44:24.744606 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 01:44:24.744606 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 01:44:24.744606 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 01:44:24.744606 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 01:44:24.744606 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application with Lock( Aug 30 01:44:24.744606 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 01:44:24.744606 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._enter() Aug 30 01:44:24.744606 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 01:44:24.744606 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 01:44:24.744606 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 01:44:24.744606 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.creator() Aug 30 01:44:24.744606 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 01:44:24.744606 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application created_value = creator( Aug 30 01:44:24.745554 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 30 01:44:24.745554 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 30 01:44:24.745554 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 30 01:44:24.745554 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 30 01:44:24.745554 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 30 01:44:24.745554 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 30 01:44:24.745554 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 30 01:44:24.745554 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application Aug 30 01:44:24.745554 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 533/1076] 10.222.0.27 () {56 vars in 1111 bytes} [Tue Aug 30 01:44:24 2022] GET /identity/v3/projects/service => generated 88 bytes in 159 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 01:44:24.750304 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4545918c-94b4-48a7-b73d-5b3a74007e3a None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:44:24.761795 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-18cfd0fc-e854-4e82-992a-da794f76b1df None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 01:44:24.762968 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-18cfd0fc-e854-4e82-992a-da794f76b1df None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:44:24.795317 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:44:24.827075 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:44:24.832621 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 544/1077] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Aug 30 01:44:24 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Aug 30 01:44:24.849393 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4545918c-94b4-48a7-b73d-5b3a74007e3a None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:44:24.882303 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15c1bbe2-4537-48f7-af75-e6957c273472 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:44:24.882895 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15c1bbe2-4537-48f7-af75-e6957c273472 None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:44:24.883167 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15c1bbe2-4537-48f7-af75-e6957c273472 None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:44:24.883434 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15c1bbe2-4537-48f7-af75-e6957c273472 None admin] PATH_INFO: `/v3/projects` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:44:24.884107 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15c1bbe2-4537-48f7-af75-e6957c273472 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81578) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:44:24.884408 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15c1bbe2-4537-48f7-af75-e6957c273472 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:44:24.885830 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15c1bbe2-4537-48f7-af75-e6957c273472 None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:44:24.902130 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 534/1078] 10.222.0.27 () {56 vars in 1148 bytes} [Tue Aug 30 01:44:24 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 30 01:44:24.908069 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18cfd0fc-e854-4e82-992a-da794f76b1df None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:44:25.004767 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18cfd0fc-e854-4e82-992a-da794f76b1df None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:44:25.038228 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbfdd689-c32b-4feb-9bca-50a00e29b9ec None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:44:25.038875 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbfdd689-c32b-4feb-9bca-50a00e29b9ec None admin] REQUEST_METHOD: `PUT` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:44:25.039149 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbfdd689-c32b-4feb-9bca-50a00e29b9ec None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:44:25.039414 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbfdd689-c32b-4feb-9bca-50a00e29b9ec None admin] PATH_INFO: `/v3/projects/4562f3951fbe465280073e60a11e44e2/users/d64f2e71361644589bec8a06e6eceed1/roles/8e1192374afc4af6a7cea01c69e72402` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:44:25.041555 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbfdd689-c32b-4feb-9bca-50a00e29b9ec None admin] RBAC: Authorizing `identity:create_grant(project_id=4562f3951fbe465280073e60a11e44e2, user_id=d64f2e71361644589bec8a06e6eceed1, role_id=8e1192374afc4af6a7cea01c69e72402)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:44:25.042968 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbfdd689-c32b-4feb-9bca-50a00e29b9ec None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:44:25.107363 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15c1bbe2-4537-48f7-af75-e6957c273472 None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:44:25.120355 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 545/1079] 10.222.0.27 () {58 vars in 1439 bytes} [Tue Aug 30 01:44:24 2022] PUT /identity/v3/projects/4562f3951fbe465280073e60a11e44e2/users/d64f2e71361644589bec8a06e6eceed1/roles/8e1192374afc4af6a7cea01c69e72402 => generated 0 bytes in 213 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 01:44:25.214484 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15c1bbe2-4537-48f7-af75-e6957c273472 None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:44:25.216894 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:44:25.250816 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06122807-a2f2-47b1-bc69-815d1707e97a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:44:25.251446 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06122807-a2f2-47b1-bc69-815d1707e97a None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:44:25.251721 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06122807-a2f2-47b1-bc69-815d1707e97a None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:44:25.251988 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06122807-a2f2-47b1-bc69-815d1707e97a None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:44:25.264557 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06122807-a2f2-47b1-bc69-815d1707e97a None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 30 01:44:25.269936 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application [None req-06122807-a2f2-47b1-bc69-815d1707e97a None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 30 01:44:25.269936 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 01:44:25.269936 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 01:44:25.269936 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 01:44:25.269936 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 01:44:25.269936 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 01:44:25.269936 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 01:44:25.269936 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 01:44:25.269936 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 01:44:25.269936 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 01:44:25.269936 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 01:44:25.269936 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 01:44:25.269936 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 30 01:44:25.269936 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 30 01:44:25.269936 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 30 01:44:25.269936 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 30 01:44:25.271086 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 30 01:44:25.271086 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 30 01:44:25.271086 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 30 01:44:25.271086 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 30 01:44:25.271086 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 01:44:25.271086 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 01:44:25.271086 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 01:44:25.271086 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 01:44:25.271086 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 01:44:25.271086 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 01:44:25.271086 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 01:44:25.271086 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application with Lock( Aug 30 01:44:25.271086 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 01:44:25.271086 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._enter() Aug 30 01:44:25.271086 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 01:44:25.271086 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 01:44:25.272287 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 01:44:25.272287 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.creator() Aug 30 01:44:25.272287 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 01:44:25.272287 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application created_value = creator( Aug 30 01:44:25.272287 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 30 01:44:25.272287 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 30 01:44:25.272287 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 30 01:44:25.272287 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 30 01:44:25.272287 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 30 01:44:25.272287 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 30 01:44:25.272287 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 30 01:44:25.272287 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application Aug 30 01:44:25.272287 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 535/1080] 10.222.0.27 () {56 vars in 1096 bytes} [Tue Aug 30 01:44:25 2022] GET /identity/v3/roles/admin => generated 83 bytes in 164 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 01:44:25.277113 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbfdd689-c32b-4feb-9bca-50a00e29b9ec None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:44:25.335685 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:44:25.376576 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbfdd689-c32b-4feb-9bca-50a00e29b9ec None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:44:25.412270 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-886e91ae-0bdf-48b3-b276-a3165b583bdc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:44:25.413048 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-886e91ae-0bdf-48b3-b276-a3165b583bdc None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:44:25.413383 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-886e91ae-0bdf-48b3-b276-a3165b583bdc None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:44:25.413712 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-886e91ae-0bdf-48b3-b276-a3165b583bdc None admin] PATH_INFO: `/v3/roles` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:44:25.414387 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-886e91ae-0bdf-48b3-b276-a3165b583bdc None admin] RBAC: Adding query filter params (name=admin) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:44:25.414694 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-886e91ae-0bdf-48b3-b276-a3165b583bdc None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:44:25.416067 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-886e91ae-0bdf-48b3-b276-a3165b583bdc None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:44:25.431051 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 546/1081] 10.222.0.27 () {56 vars in 1099 bytes} [Tue Aug 30 01:44:25 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:44:25.436818 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06122807-a2f2-47b1-bc69-815d1707e97a None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:44:25.537214 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06122807-a2f2-47b1-bc69-815d1707e97a None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:44:25.570698 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-711a505e-abdd-469f-bddc-e4bf4ae43347 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:44:25.571324 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-711a505e-abdd-469f-bddc-e4bf4ae43347 None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:44:25.571604 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-711a505e-abdd-469f-bddc-e4bf4ae43347 None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:44:25.571890 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-711a505e-abdd-469f-bddc-e4bf4ae43347 None admin] PATH_INFO: `/v3/users/7ce2deb93a6249aa96a701b6bd690844` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:44:25.572981 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-711a505e-abdd-469f-bddc-e4bf4ae43347 None admin] RBAC: Authorizing `identity:get_user(user_id=7ce2deb93a6249aa96a701b6bd690844)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:44:25.574442 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-711a505e-abdd-469f-bddc-e4bf4ae43347 None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:44:25.580713 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 536/1082] 10.222.0.27 () {56 vars in 1177 bytes} [Tue Aug 30 01:44:25 2022] GET /identity/v3/users/7ce2deb93a6249aa96a701b6bd690844 => generated 285 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 30 01:44:25.582408 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-886e91ae-0bdf-48b3-b276-a3165b583bdc None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:44:25.682697 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-886e91ae-0bdf-48b3-b276-a3165b583bdc None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:44:25.704547 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:44:25.719133 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8a50dc0-37fd-4d58-8e02-610b3c1a351b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:44:25.719763 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8a50dc0-37fd-4d58-8e02-610b3c1a351b None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:44:25.720038 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8a50dc0-37fd-4d58-8e02-610b3c1a351b None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:44:25.720308 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8a50dc0-37fd-4d58-8e02-610b3c1a351b None admin] PATH_INFO: `/v3/projects/c09a62e9f672418589e36f7395117fdb` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:44:25.721422 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8a50dc0-37fd-4d58-8e02-610b3c1a351b None admin] RBAC: Authorizing `identity:get_project(project_id=c09a62e9f672418589e36f7395117fdb)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:44:25.722961 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8a50dc0-37fd-4d58-8e02-610b3c1a351b None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:44:25.730061 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 547/1083] 10.222.0.27 () {56 vars in 1186 bytes} [Tue Aug 30 01:44:25 2022] GET /identity/v3/projects/c09a62e9f672418589e36f7395117fdb => generated 311 bytes in 144 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:44:25.730998 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-711a505e-abdd-469f-bddc-e4bf4ae43347 None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:44:25.763488 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:44:25.830040 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-711a505e-abdd-469f-bddc-e4bf4ae43347 None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:44:25.863675 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-758cc2cc-269a-430f-914e-d4201cb3f916 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:44:25.864322 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-758cc2cc-269a-430f-914e-d4201cb3f916 None admin] REQUEST_METHOD: `PUT` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:44:25.864604 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-758cc2cc-269a-430f-914e-d4201cb3f916 None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:44:25.864876 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-758cc2cc-269a-430f-914e-d4201cb3f916 None admin] PATH_INFO: `/v3/projects/c09a62e9f672418589e36f7395117fdb/users/7ce2deb93a6249aa96a701b6bd690844/roles/ead5fac86a0747ea8823a82efcea8ed7` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:44:25.866797 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-758cc2cc-269a-430f-914e-d4201cb3f916 None admin] RBAC: Authorizing `identity:create_grant(project_id=c09a62e9f672418589e36f7395117fdb, user_id=7ce2deb93a6249aa96a701b6bd690844, role_id=ead5fac86a0747ea8823a82efcea8ed7)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:44:25.868216 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-758cc2cc-269a-430f-914e-d4201cb3f916 None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:44:25.936454 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 537/1084] 10.222.0.27 () {58 vars in 1439 bytes} [Tue Aug 30 01:44:25 2022] PUT /identity/v3/projects/c09a62e9f672418589e36f7395117fdb/users/7ce2deb93a6249aa96a701b6bd690844/roles/ead5fac86a0747ea8823a82efcea8ed7 => generated 0 bytes in 206 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 01:44:27.049851 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fd00de9-f058-4822-8ef1-ed74737159b3 None None] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:44:27.050944 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fd00de9-f058-4822-8ef1-ed74737159b3 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:44:27.051087 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fd00de9-f058-4822-8ef1-ed74737159b3 None None] PATH_INFO: `/` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:44:27.052432 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 548/1085] 10.222.0.27 () {54 vars in 927 bytes} [Tue Aug 30 01:44:27 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 01:44:27.057606 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec67b9ae-f54f-4fb2-8461-9e1092637420 None None] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:44:27.058026 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec67b9ae-f54f-4fb2-8461-9e1092637420 None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:44:27.058152 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec67b9ae-f54f-4fb2-8461-9e1092637420 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:44:27.075764 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.auth.core [None req-ec67b9ae-f54f-4fb2-8461-9e1092637420 None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81578) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 01:44:27.076920 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-ec67b9ae-f54f-4fb2-8461-9e1092637420 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:44:27.140830 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 538/1086] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Aug 30 01:44:27 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 30 01:44:27.409115 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fd00de9-f058-4822-8ef1-ed74737159b3 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:44:27.511219 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fd00de9-f058-4822-8ef1-ed74737159b3 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:44:27.544941 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4febd15f-476b-4c8d-b7b2-ed313bb56e3e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:44:27.545639 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4febd15f-476b-4c8d-b7b2-ed313bb56e3e None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:44:27.545927 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4febd15f-476b-4c8d-b7b2-ed313bb56e3e None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:44:27.546198 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4febd15f-476b-4c8d-b7b2-ed313bb56e3e None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:44:27.556240 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4febd15f-476b-4c8d-b7b2-ed313bb56e3e None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 30 01:44:27.560957 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application [None req-4febd15f-476b-4c8d-b7b2-ed313bb56e3e None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Aug 30 01:44:27.560957 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 01:44:27.560957 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 01:44:27.560957 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 01:44:27.560957 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 01:44:27.560957 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 01:44:27.560957 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 01:44:27.560957 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 01:44:27.560957 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 01:44:27.560957 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 01:44:27.560957 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 01:44:27.560957 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 01:44:27.560957 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 30 01:44:27.560957 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 30 01:44:27.560957 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 30 01:44:27.560957 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 30 01:44:27.562132 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 30 01:44:27.562132 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 30 01:44:27.562132 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 30 01:44:27.562132 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 30 01:44:27.562132 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 01:44:27.562132 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 01:44:27.562132 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 01:44:27.562132 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 01:44:27.562132 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 01:44:27.562132 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 01:44:27.562132 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 01:44:27.562132 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application with Lock( Aug 30 01:44:27.562132 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 01:44:27.562132 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._enter() Aug 30 01:44:27.562132 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 01:44:27.562132 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 01:44:27.563321 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 01:44:27.563321 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.creator() Aug 30 01:44:27.563321 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 01:44:27.563321 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application created_value = creator( Aug 30 01:44:27.563321 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 30 01:44:27.563321 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 30 01:44:27.563321 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 30 01:44:27.563321 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 30 01:44:27.563321 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 30 01:44:27.563321 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 30 01:44:27.563321 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Aug 30 01:44:27.563321 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Aug 30 01:44:27.563321 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 549/1087] 10.222.0.27 () {56 vars in 1120 bytes} [Tue Aug 30 01:44:27 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 154 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 01:44:27.568962 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec67b9ae-f54f-4fb2-8461-9e1092637420 None None] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:44:27.593372 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:44:27.670022 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec67b9ae-f54f-4fb2-8461-9e1092637420 None None] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:44:27.704103 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0e1dc32-b6da-443b-99e6-3d1501b0dc97 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:44:27.704770 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0e1dc32-b6da-443b-99e6-3d1501b0dc97 None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:44:27.705042 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0e1dc32-b6da-443b-99e6-3d1501b0dc97 None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:44:27.705319 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0e1dc32-b6da-443b-99e6-3d1501b0dc97 None admin] PATH_INFO: `/v3/roles` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:44:27.707237 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0e1dc32-b6da-443b-99e6-3d1501b0dc97 None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=81578) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:44:27.707237 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0e1dc32-b6da-443b-99e6-3d1501b0dc97 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:44:27.708871 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0e1dc32-b6da-443b-99e6-3d1501b0dc97 None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:44:27.717718 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 539/1088] 10.222.0.27 () {56 vars in 1115 bytes} [Tue Aug 30 01:44:27 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:44:27.723173 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4febd15f-476b-4c8d-b7b2-ed313bb56e3e None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:44:27.774778 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:44:27.824123 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4febd15f-476b-4c8d-b7b2-ed313bb56e3e None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:44:27.857622 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c745eecd-60fc-49f6-b758-3c9612d3f639 None None] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:44:27.857929 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c745eecd-60fc-49f6-b758-3c9612d3f639 None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:44:27.858165 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c745eecd-60fc-49f6-b758-3c9612d3f639 None None] PATH_INFO: `/` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:44:27.858957 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1eea1c1d-649a-43b7-9ea0-44945df707f1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:44:27.859649 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 540/1089] 10.222.0.27 () {54 vars in 927 bytes} [Tue Aug 30 01:44:27 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 01:44:27.859776 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1eea1c1d-649a-43b7-9ea0-44945df707f1 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:44:27.859867 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1eea1c1d-649a-43b7-9ea0-44945df707f1 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:44:27.860115 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1eea1c1d-649a-43b7-9ea0-44945df707f1 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:44:27.867012 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-627dc106-a3ff-454d-9c81-d85f91e6f5b1 None None] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:44:27.867441 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-627dc106-a3ff-454d-9c81-d85f91e6f5b1 None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:44:27.867858 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-627dc106-a3ff-454d-9c81-d85f91e6f5b1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:44:27.870952 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1eea1c1d-649a-43b7-9ea0-44945df707f1 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:44:27.872387 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1eea1c1d-649a-43b7-9ea0-44945df707f1 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:44:27.887496 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application [None req-1eea1c1d-649a-43b7-9ea0-44945df707f1 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 01:44:27.887496 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 01:44:27.887496 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 01:44:27.887496 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 01:44:27.887496 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 01:44:27.887496 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 01:44:27.887496 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 01:44:27.887496 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 01:44:27.887496 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 01:44:27.887496 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 01:44:27.887496 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 01:44:27.887496 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 01:44:27.887496 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 30 01:44:27.887496 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 30 01:44:27.887496 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 30 01:44:27.887496 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 30 01:44:27.888666 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 01:44:27.888666 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 01:44:27.888666 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 01:44:27.888666 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 01:44:27.888666 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 01:44:27.888666 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 01:44:27.888666 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 01:44:27.888666 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application with Lock( Aug 30 01:44:27.888666 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 01:44:27.888666 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._enter() Aug 30 01:44:27.888666 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 01:44:27.888666 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 01:44:27.888666 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 01:44:27.888666 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.creator() Aug 30 01:44:27.888666 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 01:44:27.888666 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application created_value = creator( Aug 30 01:44:27.888666 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 30 01:44:27.889607 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 30 01:44:27.889607 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 01:44:27.889607 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Aug 30 01:44:27.890062 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.auth.core [None req-627dc106-a3ff-454d-9c81-d85f91e6f5b1 None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81578) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 01:44:27.890150 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 550/1090] 10.222.0.27 () {56 vars in 1108 bytes} [Tue Aug 30 01:44:27 2022] GET /identity/v3/domains/Default => generated 87 bytes in 166 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 01:44:27.890219 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-627dc106-a3ff-454d-9c81-d85f91e6f5b1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:44:27.895183 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1eea1c1d-649a-43b7-9ea0-44945df707f1 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:44:27.961497 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 541/1091] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Aug 30 01:44:27 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Aug 30 01:44:27.991425 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:44:27.994739 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1eea1c1d-649a-43b7-9ea0-44945df707f1 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:44:28.028033 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3b44854-8ae6-4cf2-b6fe-754ba99a4a91 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:44:28.028623 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3b44854-8ae6-4cf2-b6fe-754ba99a4a91 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:44:28.028896 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3b44854-8ae6-4cf2-b6fe-754ba99a4a91 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:44:28.029161 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3b44854-8ae6-4cf2-b6fe-754ba99a4a91 None admin] PATH_INFO: `/v3/domains` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:44:28.029830 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3b44854-8ae6-4cf2-b6fe-754ba99a4a91 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:44:28.030127 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3b44854-8ae6-4cf2-b6fe-754ba99a4a91 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:44:28.031484 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3b44854-8ae6-4cf2-b6fe-754ba99a4a91 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:44:28.042511 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 551/1092] 10.222.0.27 () {56 vars in 1109 bytes} [Tue Aug 30 01:44:27 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:44:28.048195 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-627dc106-a3ff-454d-9c81-d85f91e6f5b1 None None] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:44:28.063752 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:44:28.146589 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-627dc106-a3ff-454d-9c81-d85f91e6f5b1 None None] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:44:28.197050 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c01c9c4-120f-4b72-ad93-5611b274f561 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:44:28.197840 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c01c9c4-120f-4b72-ad93-5611b274f561 None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:44:28.198171 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c01c9c4-120f-4b72-ad93-5611b274f561 None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:44:28.198493 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c01c9c4-120f-4b72-ad93-5611b274f561 None admin] PATH_INFO: `/v3/users/nova` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:44:28.206478 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c01c9c4-120f-4b72-ad93-5611b274f561 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:44:28.208131 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c01c9c4-120f-4b72-ad93-5611b274f561 None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:44:28.219767 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application [None req-4c01c9c4-120f-4b72-ad93-5611b274f561 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Aug 30 01:44:28.219767 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 01:44:28.219767 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 01:44:28.219767 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 01:44:28.219767 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 01:44:28.219767 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 01:44:28.219767 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 01:44:28.219767 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 01:44:28.219767 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 01:44:28.219767 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 01:44:28.219767 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 01:44:28.219767 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 01:44:28.219767 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 30 01:44:28.219767 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 30 01:44:28.219767 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 30 01:44:28.219767 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 30 01:44:28.220882 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 01:44:28.220882 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 01:44:28.220882 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 30 01:44:28.220882 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 30 01:44:28.220882 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 30 01:44:28.220882 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 30 01:44:28.220882 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 01:44:28.220882 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 01:44:28.220882 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 01:44:28.220882 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 01:44:28.220882 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 01:44:28.220882 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application with Lock( Aug 30 01:44:28.220882 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 01:44:28.220882 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._enter() Aug 30 01:44:28.220882 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 01:44:28.220882 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 01:44:28.220882 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 01:44:28.220882 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.creator() Aug 30 01:44:28.222405 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 01:44:28.222405 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application created_value = creator( Aug 30 01:44:28.222405 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 30 01:44:28.222405 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 30 01:44:28.222405 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 30 01:44:28.222405 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 30 01:44:28.222405 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 30 01:44:28.222405 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 30 01:44:28.222405 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Aug 30 01:44:28.222405 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application Aug 30 01:44:28.225771 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 542/1093] 10.222.0.27 () {56 vars in 1093 bytes} [Tue Aug 30 01:44:28 2022] GET /identity/v3/users/nova => generated 82 bytes in 178 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 01:44:28.227236 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3b44854-8ae6-4cf2-b6fe-754ba99a4a91 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:44:28.243531 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c01c9c4-120f-4b72-ad93-5611b274f561 None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:44:28.315851 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:44:28.335635 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3b44854-8ae6-4cf2-b6fe-754ba99a4a91 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:44:28.340596 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c01c9c4-120f-4b72-ad93-5611b274f561 None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:44:28.369419 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecb7cec6-cb7b-4cda-8671-7ce6c521abfb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:44:28.370056 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecb7cec6-cb7b-4cda-8671-7ce6c521abfb None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:44:28.370329 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecb7cec6-cb7b-4cda-8671-7ce6c521abfb None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:44:28.370602 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecb7cec6-cb7b-4cda-8671-7ce6c521abfb None admin] PATH_INFO: `/v3/users` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:44:28.371320 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ecb7cec6-cb7b-4cda-8671-7ce6c521abfb None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:44:28.371623 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ecb7cec6-cb7b-4cda-8671-7ce6c521abfb None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:44:28.373024 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ecb7cec6-cb7b-4cda-8671-7ce6c521abfb None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:44:28.375245 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b91dfe52-ae0a-4753-a6ad-d869efd8fe85 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:44:28.375869 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b91dfe52-ae0a-4753-a6ad-d869efd8fe85 None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:44:28.376145 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b91dfe52-ae0a-4753-a6ad-d869efd8fe85 None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:44:28.376411 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b91dfe52-ae0a-4753-a6ad-d869efd8fe85 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:44:28.386288 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b91dfe52-ae0a-4753-a6ad-d869efd8fe85 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 30 01:44:28.389829 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 552/1094] 10.222.0.27 () {56 vars in 1133 bytes} [Tue Aug 30 01:44:28 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:44:28.391426 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application [None req-b91dfe52-ae0a-4753-a6ad-d869efd8fe85 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 30 01:44:28.391426 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 01:44:28.391426 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 01:44:28.391426 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 01:44:28.391426 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 01:44:28.391426 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 01:44:28.391426 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 01:44:28.391426 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 01:44:28.391426 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 01:44:28.391426 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 01:44:28.391426 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 01:44:28.391426 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 01:44:28.391426 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 30 01:44:28.391426 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 30 01:44:28.391426 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 30 01:44:28.391426 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 30 01:44:28.392399 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 30 01:44:28.392399 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 30 01:44:28.392399 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 30 01:44:28.392399 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 30 01:44:28.392399 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 01:44:28.392399 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 01:44:28.392399 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 01:44:28.392399 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 01:44:28.392399 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 01:44:28.392399 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 01:44:28.392399 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 01:44:28.392399 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application with Lock( Aug 30 01:44:28.392399 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 01:44:28.392399 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._enter() Aug 30 01:44:28.392399 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 01:44:28.392399 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 01:44:28.393276 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 01:44:28.393276 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.creator() Aug 30 01:44:28.393276 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 01:44:28.393276 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application created_value = creator( Aug 30 01:44:28.393276 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 30 01:44:28.393276 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 30 01:44:28.393276 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 30 01:44:28.393276 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 30 01:44:28.393276 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 30 01:44:28.393276 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 30 01:44:28.393276 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 30 01:44:28.393276 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application Aug 30 01:44:28.393276 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 543/1095] 10.222.0.27 () {56 vars in 1096 bytes} [Tue Aug 30 01:44:28 2022] GET /identity/v3/roles/admin => generated 83 bytes in 152 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 30 01:44:28.398303 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecb7cec6-cb7b-4cda-8671-7ce6c521abfb None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:44:28.398420 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b91dfe52-ae0a-4753-a6ad-d869efd8fe85 None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:44:28.417020 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:44:28.497995 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecb7cec6-cb7b-4cda-8671-7ce6c521abfb None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:44:28.498175 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b91dfe52-ae0a-4753-a6ad-d869efd8fe85 None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:44:28.534201 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e172dff9-21b1-4b31-8d75-6c7d1cf68ae2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:44:28.535470 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e172dff9-21b1-4b31-8d75-6c7d1cf68ae2 None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:44:28.535747 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e172dff9-21b1-4b31-8d75-6c7d1cf68ae2 None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:44:28.536012 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e172dff9-21b1-4b31-8d75-6c7d1cf68ae2 None admin] PATH_INFO: `/v3/roles` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:44:28.537071 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e172dff9-21b1-4b31-8d75-6c7d1cf68ae2 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81578) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:44:28.537502 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e172dff9-21b1-4b31-8d75-6c7d1cf68ae2 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:44:28.539509 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e172dff9-21b1-4b31-8d75-6c7d1cf68ae2 None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:44:28.539738 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8851fb8e-e287-4ad6-925d-90f61ab8d819 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:44:28.540345 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8851fb8e-e287-4ad6-925d-90f61ab8d819 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:44:28.540622 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8851fb8e-e287-4ad6-925d-90f61ab8d819 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:44:28.540976 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8851fb8e-e287-4ad6-925d-90f61ab8d819 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:44:28.541084 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:44:28.549016 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 544/1096] 10.222.0.27 () {56 vars in 1099 bytes} [Tue Aug 30 01:44:28 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 30 01:44:28.555609 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e172dff9-21b1-4b31-8d75-6c7d1cf68ae2 None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:44:28.608905 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8851fb8e-e287-4ad6-925d-90f61ab8d819 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:44:28.610414 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8851fb8e-e287-4ad6-925d-90f61ab8d819 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:44:28.629010 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:44:28.653960 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e172dff9-21b1-4b31-8d75-6c7d1cf68ae2 None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:44:28.688815 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88691f59-c48c-4c63-b27b-68446fe6b921 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:44:28.689444 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88691f59-c48c-4c63-b27b-68446fe6b921 None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:44:28.689753 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88691f59-c48c-4c63-b27b-68446fe6b921 None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:44:28.690076 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88691f59-c48c-4c63-b27b-68446fe6b921 None admin] PATH_INFO: `/v3/users/7ce2deb93a6249aa96a701b6bd690844` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:44:28.698113 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88691f59-c48c-4c63-b27b-68446fe6b921 None admin] RBAC: Authorizing `identity:get_user(user_id=7ce2deb93a6249aa96a701b6bd690844)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:44:28.698113 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88691f59-c48c-4c63-b27b-68446fe6b921 None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:44:28.698113 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 545/1097] 10.222.0.27 () {56 vars in 1177 bytes} [Tue Aug 30 01:44:28 2022] GET /identity/v3/users/7ce2deb93a6249aa96a701b6bd690844 => generated 285 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 30 01:44:28.700270 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88691f59-c48c-4c63-b27b-68446fe6b921 None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:44:28.712027 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 553/1098] 10.222.0.27 () {58 vars in 1282 bytes} [Tue Aug 30 01:44:28 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 314 msecs (HTTP/1.1 200) 6 headers in 359 bytes (2 switches on core 0) Aug 30 01:44:28.717672 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8851fb8e-e287-4ad6-925d-90f61ab8d819 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:44:28.772049 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:44:28.808338 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88691f59-c48c-4c63-b27b-68446fe6b921 None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:44:28.815600 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8851fb8e-e287-4ad6-925d-90f61ab8d819 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:44:28.832488 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:44:28.842825 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d1eef19-2923-45a9-af8f-79174528c6d2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:44:28.843427 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d1eef19-2923-45a9-af8f-79174528c6d2 None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:44:28.843708 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d1eef19-2923-45a9-af8f-79174528c6d2 None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:44:28.843977 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d1eef19-2923-45a9-af8f-79174528c6d2 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:44:28.849216 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3426be24-aa89-4944-857b-ea0463e602ef None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:44:28.849866 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3426be24-aa89-4944-857b-ea0463e602ef None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:44:28.850147 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3426be24-aa89-4944-857b-ea0463e602ef None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:44:28.850416 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3426be24-aa89-4944-857b-ea0463e602ef None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:44:28.861200 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3426be24-aa89-4944-857b-ea0463e602ef None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:44:28.863197 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3426be24-aa89-4944-857b-ea0463e602ef None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:44:28.877009 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application [None req-3426be24-aa89-4944-857b-ea0463e602ef None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 01:44:28.877009 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 01:44:28.877009 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 01:44:28.877009 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 01:44:28.877009 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 01:44:28.877009 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 01:44:28.877009 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 01:44:28.877009 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 01:44:28.877009 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 01:44:28.877009 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 01:44:28.877009 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 01:44:28.877009 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 01:44:28.877009 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 30 01:44:28.877009 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 30 01:44:28.877009 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 30 01:44:28.877009 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 30 01:44:28.878353 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 01:44:28.878353 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 01:44:28.878353 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 01:44:28.878353 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 01:44:28.878353 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 01:44:28.878353 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 01:44:28.878353 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 01:44:28.878353 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application with Lock( Aug 30 01:44:28.878353 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 01:44:28.878353 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._enter() Aug 30 01:44:28.878353 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 01:44:28.878353 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 01:44:28.878353 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 01:44:28.878353 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.creator() Aug 30 01:44:28.878353 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 01:44:28.878353 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application created_value = creator( Aug 30 01:44:28.878353 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 30 01:44:28.879278 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 30 01:44:28.879278 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 01:44:28.879278 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Aug 30 01:44:28.879278 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 554/1099] 10.222.0.27 () {56 vars in 1108 bytes} [Tue Aug 30 01:44:28 2022] GET /identity/v3/domains/Default => generated 87 bytes in 161 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 30 01:44:28.883797 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3426be24-aa89-4944-857b-ea0463e602ef None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:44:28.916253 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d1eef19-2923-45a9-af8f-79174528c6d2 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:44:28.917722 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d1eef19-2923-45a9-af8f-79174528c6d2 None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:44:28.976629 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:44:28.985038 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3426be24-aa89-4944-857b-ea0463e602ef None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:44:29.014347 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 546/1100] 10.222.0.27 () {58 vars in 1282 bytes} [Tue Aug 30 01:44:28 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 314 msecs (HTTP/1.1 200) 6 headers in 359 bytes (2 switches on core 0) Aug 30 01:44:29.017771 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c8bac45-58ab-420e-8791-145159ad53e2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:44:29.018368 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c8bac45-58ab-420e-8791-145159ad53e2 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:44:29.018643 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c8bac45-58ab-420e-8791-145159ad53e2 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:44:29.019462 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c8bac45-58ab-420e-8791-145159ad53e2 None admin] PATH_INFO: `/v3/domains` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:44:29.020101 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c8bac45-58ab-420e-8791-145159ad53e2 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:44:29.020404 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c8bac45-58ab-420e-8791-145159ad53e2 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:44:29.020959 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d1eef19-2923-45a9-af8f-79174528c6d2 None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:44:29.021819 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c8bac45-58ab-420e-8791-145159ad53e2 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:44:29.033833 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 555/1101] 10.222.0.27 () {56 vars in 1109 bytes} [Tue Aug 30 01:44:28 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 30 01:44:29.040010 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c8bac45-58ab-420e-8791-145159ad53e2 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:44:29.069179 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:44:29.119580 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d1eef19-2923-45a9-af8f-79174528c6d2 None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:44:29.137639 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c8bac45-58ab-420e-8791-145159ad53e2 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:44:29.152351 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e57c15c0-72b4-4206-b8e8-00fef0b47a9a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:44:29.152963 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e57c15c0-72b4-4206-b8e8-00fef0b47a9a None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:44:29.153236 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e57c15c0-72b4-4206-b8e8-00fef0b47a9a None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:44:29.153526 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e57c15c0-72b4-4206-b8e8-00fef0b47a9a None admin] PATH_INFO: `/v3/projects/c09a62e9f672418589e36f7395117fdb` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:44:29.154618 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e57c15c0-72b4-4206-b8e8-00fef0b47a9a None admin] RBAC: Authorizing `identity:get_project(project_id=c09a62e9f672418589e36f7395117fdb)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:44:29.156019 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e57c15c0-72b4-4206-b8e8-00fef0b47a9a None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:44:29.157647 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 547/1102] 10.222.0.27 () {56 vars in 1186 bytes} [Tue Aug 30 01:44:29 2022] GET /identity/v3/projects/c09a62e9f672418589e36f7395117fdb => generated 311 bytes in 137 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 30 01:44:29.164295 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e57c15c0-72b4-4206-b8e8-00fef0b47a9a None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:44:29.171341 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffabda92-e391-41f2-a05c-1a3c04b922d1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:44:29.171972 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffabda92-e391-41f2-a05c-1a3c04b922d1 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:44:29.172250 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffabda92-e391-41f2-a05c-1a3c04b922d1 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:44:29.172526 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffabda92-e391-41f2-a05c-1a3c04b922d1 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:44:29.180070 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffabda92-e391-41f2-a05c-1a3c04b922d1 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:44:29.181538 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffabda92-e391-41f2-a05c-1a3c04b922d1 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:44:29.192528 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application [None req-ffabda92-e391-41f2-a05c-1a3c04b922d1 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 30 01:44:29.192528 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 01:44:29.192528 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 01:44:29.192528 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 01:44:29.192528 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 01:44:29.192528 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 01:44:29.192528 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 01:44:29.192528 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 01:44:29.192528 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 01:44:29.192528 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 01:44:29.192528 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 01:44:29.192528 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 01:44:29.192528 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 30 01:44:29.192528 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 30 01:44:29.192528 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 30 01:44:29.193441 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 30 01:44:29.193441 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 01:44:29.193441 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 01:44:29.193441 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 01:44:29.193441 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 01:44:29.193441 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 01:44:29.193441 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 01:44:29.193441 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 01:44:29.193441 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application with Lock( Aug 30 01:44:29.193441 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 01:44:29.193441 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._enter() Aug 30 01:44:29.193441 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 01:44:29.193441 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 01:44:29.193441 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 01:44:29.193441 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.creator() Aug 30 01:44:29.193441 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 01:44:29.193441 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application created_value = creator( Aug 30 01:44:29.194481 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 30 01:44:29.194481 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 30 01:44:29.194481 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 30 01:44:29.194481 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 30 01:44:29.194481 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 30 01:44:29.194481 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 30 01:44:29.194481 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 30 01:44:29.194481 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Aug 30 01:44:29.197268 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:44:29.197745 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 556/1103] 10.222.0.27 () {56 vars in 1111 bytes} [Tue Aug 30 01:44:29 2022] GET /identity/v3/projects/service => generated 88 bytes in 158 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 30 01:44:29.199561 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffabda92-e391-41f2-a05c-1a3c04b922d1 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:44:29.262517 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e57c15c0-72b4-4206-b8e8-00fef0b47a9a None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:44:29.288060 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:44:29.296224 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55f3df39-e39f-473e-91d7-151d2ee82c8d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:44:29.298880 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55f3df39-e39f-473e-91d7-151d2ee82c8d None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:44:29.298880 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55f3df39-e39f-473e-91d7-151d2ee82c8d None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:44:29.298880 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55f3df39-e39f-473e-91d7-151d2ee82c8d None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:44:29.299108 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffabda92-e391-41f2-a05c-1a3c04b922d1 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:44:29.299196 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55f3df39-e39f-473e-91d7-151d2ee82c8d None admin] RBAC: Adding query filter params (role.id=ead5fac86a0747ea8823a82efcea8ed7, scope.project.id=c09a62e9f672418589e36f7395117fdb, user.id=7ce2deb93a6249aa96a701b6bd690844) {{(pid=81578) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:44:29.299196 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55f3df39-e39f-473e-91d7-151d2ee82c8d None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:44:29.299996 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55f3df39-e39f-473e-91d7-151d2ee82c8d None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:44:29.306223 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 548/1104] 10.222.0.27 () {56 vars in 1374 bytes} [Tue Aug 30 01:44:29 2022] GET /identity/v3/role_assignments?user.id=7ce2deb93a6249aa96a701b6bd690844&scope.project.id=c09a62e9f672418589e36f7395117fdb&role.id=ead5fac86a0747ea8823a82efcea8ed7 => generated 608 bytes in 142 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 30 01:44:29.334196 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a2aa2a4-0ee2-4c85-827e-540726c0b025 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:44:29.334814 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a2aa2a4-0ee2-4c85-827e-540726c0b025 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:44:29.335094 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a2aa2a4-0ee2-4c85-827e-540726c0b025 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:44:29.335365 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a2aa2a4-0ee2-4c85-827e-540726c0b025 None admin] PATH_INFO: `/v3/projects` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:44:29.336078 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a2aa2a4-0ee2-4c85-827e-540726c0b025 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:44:29.336382 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a2aa2a4-0ee2-4c85-827e-540726c0b025 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:44:29.337886 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a2aa2a4-0ee2-4c85-827e-540726c0b025 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:44:29.351892 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 557/1105] 10.222.0.27 () {56 vars in 1148 bytes} [Tue Aug 30 01:44:29 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 30 01:44:29.358345 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55f3df39-e39f-473e-91d7-151d2ee82c8d None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:44:29.459033 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:44:29.464754 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55f3df39-e39f-473e-91d7-151d2ee82c8d None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:44:29.500292 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59a36e24-7c3c-40b1-b79a-7aa3189931d6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:44:29.500904 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59a36e24-7c3c-40b1-b79a-7aa3189931d6 None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:44:29.501182 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59a36e24-7c3c-40b1-b79a-7aa3189931d6 None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:44:29.501477 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59a36e24-7c3c-40b1-b79a-7aa3189931d6 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:44:29.502197 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59a36e24-7c3c-40b1-b79a-7aa3189931d6 None admin] RBAC: Adding query filter params (role.id=8e1192374afc4af6a7cea01c69e72402, scope.project.id=4562f3951fbe465280073e60a11e44e2, user.id=d64f2e71361644589bec8a06e6eceed1) {{(pid=81578) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:44:29.502509 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59a36e24-7c3c-40b1-b79a-7aa3189931d6 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:44:29.504477 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59a36e24-7c3c-40b1-b79a-7aa3189931d6 None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:44:29.513073 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 549/1106] 10.222.0.27 () {56 vars in 1374 bytes} [Tue Aug 30 01:44:29 2022] GET /identity/v3/role_assignments?user.id=d64f2e71361644589bec8a06e6eceed1&scope.project.id=4562f3951fbe465280073e60a11e44e2&role.id=8e1192374afc4af6a7cea01c69e72402 => generated 608 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:44:31.293372 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa19f50b-50ec-41a2-bada-db587de113ea None None] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:44:31.293931 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa19f50b-50ec-41a2-bada-db587de113ea None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:44:31.293931 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa19f50b-50ec-41a2-bada-db587de113ea None None] PATH_INFO: `/` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:44:31.295275 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 558/1107] 10.222.0.27 () {54 vars in 927 bytes} [Tue Aug 30 01:44:31 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 01:44:31.300420 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f2cf441-7c9d-4dd4-b723-8a1106bc9b62 None None] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:44:31.300845 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f2cf441-7c9d-4dd4-b723-8a1106bc9b62 None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:44:31.300974 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f2cf441-7c9d-4dd4-b723-8a1106bc9b62 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:44:31.319265 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.auth.core [None req-5f2cf441-7c9d-4dd4-b723-8a1106bc9b62 None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81578) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 01:44:31.320424 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-5f2cf441-7c9d-4dd4-b723-8a1106bc9b62 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:44:31.384686 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 550/1108] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Aug 30 01:44:31 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 30 01:44:31.655294 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa19f50b-50ec-41a2-bada-db587de113ea None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:44:31.755971 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa19f50b-50ec-41a2-bada-db587de113ea None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:44:31.788181 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f220b47-a027-488a-bad0-cfbc3f2b0bca None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:44:31.788796 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f220b47-a027-488a-bad0-cfbc3f2b0bca None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:44:31.789073 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f220b47-a027-488a-bad0-cfbc3f2b0bca None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:44:31.789347 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f220b47-a027-488a-bad0-cfbc3f2b0bca None admin] PATH_INFO: `/v3/domains/default` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:44:31.790468 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f220b47-a027-488a-bad0-cfbc3f2b0bca None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:44:31.791893 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f220b47-a027-488a-bad0-cfbc3f2b0bca None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:44:31.793468 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 559/1109] 10.222.0.27 () {56 vars in 1108 bytes} [Tue Aug 30 01:44:31 2022] GET /identity/v3/domains/default => generated 199 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:44:31.799197 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f2cf441-7c9d-4dd4-b723-8a1106bc9b62 None None] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:44:31.887880 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:44:31.897749 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f2cf441-7c9d-4dd4-b723-8a1106bc9b62 None None] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:44:31.932535 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aabd6806-e5cf-4c3e-a0e0-237fb6c3b7f4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:44:31.933060 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aabd6806-e5cf-4c3e-a0e0-237fb6c3b7f4 None admin] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:44:31.933337 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aabd6806-e5cf-4c3e-a0e0-237fb6c3b7f4 None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:44:31.933636 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aabd6806-e5cf-4c3e-a0e0-237fb6c3b7f4 None admin] PATH_INFO: `/v3/users` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:44:31.934293 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aabd6806-e5cf-4c3e-a0e0-237fb6c3b7f4 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:44:31.935707 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aabd6806-e5cf-4c3e-a0e0-237fb6c3b7f4 None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:44:32.016423 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 551/1110] 10.222.0.27 () {60 vars in 1132 bytes} [Tue Aug 30 01:44:31 2022] POST /identity/v3/users => generated 286 bytes in 218 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 01:44:33.945357 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32e41e08-a09a-4cc5-9d95-0403865a0bc5 None None] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:44:33.945964 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32e41e08-a09a-4cc5-9d95-0403865a0bc5 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:44:33.945964 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32e41e08-a09a-4cc5-9d95-0403865a0bc5 None None] PATH_INFO: `/` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:44:33.947282 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 560/1111] 10.222.0.27 () {54 vars in 927 bytes} [Tue Aug 30 01:44:33 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 01:44:33.952323 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f94ab94-0fab-4dc3-8546-0eb7cb761fc5 None None] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:44:33.952729 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f94ab94-0fab-4dc3-8546-0eb7cb761fc5 None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:44:33.952871 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f94ab94-0fab-4dc3-8546-0eb7cb761fc5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:44:33.971399 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.auth.core [None req-5f94ab94-0fab-4dc3-8546-0eb7cb761fc5 None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81578) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 01:44:33.973079 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-5f94ab94-0fab-4dc3-8546-0eb7cb761fc5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:44:34.037586 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 552/1112] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Aug 30 01:44:33 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 30 01:44:34.307838 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32e41e08-a09a-4cc5-9d95-0403865a0bc5 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:44:34.327143 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:44:34.407906 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32e41e08-a09a-4cc5-9d95-0403865a0bc5 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:44:34.442072 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a5ae83e-b8b1-42e2-a1d0-7063a3344f8c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:44:34.442681 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a5ae83e-b8b1-42e2-a1d0-7063a3344f8c None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:44:34.442956 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a5ae83e-b8b1-42e2-a1d0-7063a3344f8c None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:44:34.443221 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a5ae83e-b8b1-42e2-a1d0-7063a3344f8c None admin] PATH_INFO: `/v3/roles/4621068972ba4f79bad104f956db04aa` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:44:34.444591 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a5ae83e-b8b1-42e2-a1d0-7063a3344f8c None admin] RBAC: Authorizing `identity:get_role(role_id=4621068972ba4f79bad104f956db04aa)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:44:34.445983 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a5ae83e-b8b1-42e2-a1d0-7063a3344f8c None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:44:34.447287 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 561/1113] 10.222.0.27 () {56 vars in 1177 bytes} [Tue Aug 30 01:44:34 2022] GET /identity/v3/roles/4621068972ba4f79bad104f956db04aa => generated 223 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:44:34.453029 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f94ab94-0fab-4dc3-8546-0eb7cb761fc5 None None] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:44:34.528426 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:44:34.551880 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f94ab94-0fab-4dc3-8546-0eb7cb761fc5 None None] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:44:34.586723 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f602de6-fbba-4ad5-859e-20f6c2736b50 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:44:34.587279 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f602de6-fbba-4ad5-859e-20f6c2736b50 None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:44:34.587557 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f602de6-fbba-4ad5-859e-20f6c2736b50 None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:44:34.587828 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f602de6-fbba-4ad5-859e-20f6c2736b50 None admin] PATH_INFO: `/v3/users/e4ff09350e5040db91f9a60c6577f3e4` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:44:34.601024 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f602de6-fbba-4ad5-859e-20f6c2736b50 None admin] RBAC: Authorizing `identity:get_user(user_id=e4ff09350e5040db91f9a60c6577f3e4)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:44:34.602743 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f602de6-fbba-4ad5-859e-20f6c2736b50 None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:44:34.604332 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 553/1114] 10.222.0.27 () {56 vars in 1177 bytes} [Tue Aug 30 01:44:34 2022] GET /identity/v3/users/e4ff09350e5040db91f9a60c6577f3e4 => generated 286 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:44:34.610449 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a5ae83e-b8b1-42e2-a1d0-7063a3344f8c None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:44:34.694466 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:44:34.709474 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a5ae83e-b8b1-42e2-a1d0-7063a3344f8c None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:44:34.744526 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e8f2b01-b20f-4967-a222-10aba282ad1b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:44:34.749562 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e8f2b01-b20f-4967-a222-10aba282ad1b None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:44:34.749562 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e8f2b01-b20f-4967-a222-10aba282ad1b None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:44:34.749562 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e8f2b01-b20f-4967-a222-10aba282ad1b None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:44:34.811187 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e8f2b01-b20f-4967-a222-10aba282ad1b None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:44:34.812694 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e8f2b01-b20f-4967-a222-10aba282ad1b None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:44:34.894413 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:44:34.909800 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 562/1115] 10.222.0.27 () {58 vars in 1282 bytes} [Tue Aug 30 01:44:34 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 300 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Aug 30 01:44:34.915963 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f602de6-fbba-4ad5-859e-20f6c2736b50 None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:44:35.062490 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f602de6-fbba-4ad5-859e-20f6c2736b50 None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:44:35.094643 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3eb35946-03f2-4842-b3d8-095f264ef918 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:44:35.095258 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3eb35946-03f2-4842-b3d8-095f264ef918 None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:44:35.095535 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3eb35946-03f2-4842-b3d8-095f264ef918 None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:44:35.095804 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3eb35946-03f2-4842-b3d8-095f264ef918 None admin] PATH_INFO: `/v3/projects/c09a62e9f672418589e36f7395117fdb` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:44:35.097434 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3eb35946-03f2-4842-b3d8-095f264ef918 None admin] RBAC: Authorizing `identity:get_project(project_id=c09a62e9f672418589e36f7395117fdb)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:44:35.098947 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3eb35946-03f2-4842-b3d8-095f264ef918 None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:44:35.100548 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 554/1116] 10.222.0.27 () {56 vars in 1186 bytes} [Tue Aug 30 01:44:34 2022] GET /identity/v3/projects/c09a62e9f672418589e36f7395117fdb => generated 311 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:44:35.106843 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e8f2b01-b20f-4967-a222-10aba282ad1b None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:44:35.204402 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e8f2b01-b20f-4967-a222-10aba282ad1b None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:44:35.238200 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92f45fce-e923-434a-ba93-1e025b9f18be None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:44:35.238819 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92f45fce-e923-434a-ba93-1e025b9f18be None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:44:35.239097 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92f45fce-e923-434a-ba93-1e025b9f18be None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:44:35.239367 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92f45fce-e923-434a-ba93-1e025b9f18be None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:44:35.240078 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92f45fce-e923-434a-ba93-1e025b9f18be None admin] RBAC: Adding query filter params (role.id=4621068972ba4f79bad104f956db04aa, scope.project.id=c09a62e9f672418589e36f7395117fdb, user.id=e4ff09350e5040db91f9a60c6577f3e4) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:44:35.240384 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92f45fce-e923-434a-ba93-1e025b9f18be None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:44:35.241818 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92f45fce-e923-434a-ba93-1e025b9f18be None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:44:35.247009 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 563/1117] 10.222.0.27 () {56 vars in 1374 bytes} [Tue Aug 30 01:44:35 2022] GET /identity/v3/role_assignments?user.id=e4ff09350e5040db91f9a60c6577f3e4&scope.project.id=c09a62e9f672418589e36f7395117fdb&role.id=4621068972ba4f79bad104f956db04aa => generated 259 bytes in 141 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:44:37.242594 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01aae101-3e6e-4a2a-b11d-5cf595943d6f None None] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:44:37.243131 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01aae101-3e6e-4a2a-b11d-5cf595943d6f None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:44:37.243223 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01aae101-3e6e-4a2a-b11d-5cf595943d6f None None] PATH_INFO: `/` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:44:37.244933 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 555/1118] 10.222.0.27 () {54 vars in 927 bytes} [Tue Aug 30 01:44:37 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 01:44:37.250389 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf0f7626-2002-477c-9e82-330cbeaf9c1c None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:44:37.250986 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf0f7626-2002-477c-9e82-330cbeaf9c1c None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:44:37.250986 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf0f7626-2002-477c-9e82-330cbeaf9c1c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:44:37.270189 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-cf0f7626-2002-477c-9e82-330cbeaf9c1c None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 01:44:37.272261 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-cf0f7626-2002-477c-9e82-330cbeaf9c1c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:44:37.323592 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:44:37.346155 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 564/1119] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Aug 30 01:44:37 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 30 01:44:37.615750 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01aae101-3e6e-4a2a-b11d-5cf595943d6f None None] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:44:37.618650 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:44:37.716944 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01aae101-3e6e-4a2a-b11d-5cf595943d6f None None] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:44:37.750546 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27d33a48-5512-4b71-96ff-261406ff12ac None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:44:37.751163 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27d33a48-5512-4b71-96ff-261406ff12ac None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:44:37.751435 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27d33a48-5512-4b71-96ff-261406ff12ac None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:44:37.751701 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27d33a48-5512-4b71-96ff-261406ff12ac None admin] PATH_INFO: `/v3/roles/4621068972ba4f79bad104f956db04aa` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:44:37.753842 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27d33a48-5512-4b71-96ff-261406ff12ac None admin] RBAC: Authorizing `identity:get_role(role_id=4621068972ba4f79bad104f956db04aa)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:44:37.755186 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27d33a48-5512-4b71-96ff-261406ff12ac None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:44:37.756497 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 556/1120] 10.222.0.27 () {56 vars in 1177 bytes} [Tue Aug 30 01:44:37 2022] GET /identity/v3/roles/4621068972ba4f79bad104f956db04aa => generated 223 bytes in 141 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:44:37.762331 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf0f7626-2002-477c-9e82-330cbeaf9c1c None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:44:37.857778 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf0f7626-2002-477c-9e82-330cbeaf9c1c None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:44:37.891291 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79179297-269e-4674-8f1f-ffa01cf37bdc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:44:37.891845 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79179297-269e-4674-8f1f-ffa01cf37bdc None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:44:37.892122 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79179297-269e-4674-8f1f-ffa01cf37bdc None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:44:37.892388 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79179297-269e-4674-8f1f-ffa01cf37bdc None admin] PATH_INFO: `/v3/users/e4ff09350e5040db91f9a60c6577f3e4` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:44:37.893496 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79179297-269e-4674-8f1f-ffa01cf37bdc None admin] RBAC: Authorizing `identity:get_user(user_id=e4ff09350e5040db91f9a60c6577f3e4)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:44:37.895074 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79179297-269e-4674-8f1f-ffa01cf37bdc None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:44:37.896649 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 565/1121] 10.222.0.27 () {56 vars in 1177 bytes} [Tue Aug 30 01:44:37 2022] GET /identity/v3/users/e4ff09350e5040db91f9a60c6577f3e4 => generated 286 bytes in 135 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:44:37.902329 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27d33a48-5512-4b71-96ff-261406ff12ac None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:44:37.970786 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:44:38.001074 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27d33a48-5512-4b71-96ff-261406ff12ac None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:44:38.033742 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b09db0ca-94e2-480f-8b15-6db75c66bf36 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:44:38.034975 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b09db0ca-94e2-480f-8b15-6db75c66bf36 None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:44:38.035243 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b09db0ca-94e2-480f-8b15-6db75c66bf36 None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:44:38.035507 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b09db0ca-94e2-480f-8b15-6db75c66bf36 None admin] PATH_INFO: `/v3/projects/c09a62e9f672418589e36f7395117fdb` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:44:38.036632 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b09db0ca-94e2-480f-8b15-6db75c66bf36 None admin] RBAC: Authorizing `identity:get_project(project_id=c09a62e9f672418589e36f7395117fdb)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:44:38.038083 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b09db0ca-94e2-480f-8b15-6db75c66bf36 None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:44:38.039690 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 557/1122] 10.222.0.27 () {56 vars in 1186 bytes} [Tue Aug 30 01:44:37 2022] GET /identity/v3/projects/c09a62e9f672418589e36f7395117fdb => generated 311 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:44:38.045690 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79179297-269e-4674-8f1f-ffa01cf37bdc None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:44:38.094797 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:44:38.144483 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79179297-269e-4674-8f1f-ffa01cf37bdc None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:44:38.177201 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e74ce1aa-f41d-4296-9323-1ddfe5ff2a55 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:44:38.178660 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e74ce1aa-f41d-4296-9323-1ddfe5ff2a55 None admin] REQUEST_METHOD: `PUT` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:44:38.178937 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e74ce1aa-f41d-4296-9323-1ddfe5ff2a55 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:44:38.179208 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e74ce1aa-f41d-4296-9323-1ddfe5ff2a55 None admin] PATH_INFO: `/v3/projects/c09a62e9f672418589e36f7395117fdb/users/e4ff09350e5040db91f9a60c6577f3e4/roles/4621068972ba4f79bad104f956db04aa` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:44:38.181318 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e74ce1aa-f41d-4296-9323-1ddfe5ff2a55 None admin] RBAC: Authorizing `identity:create_grant(project_id=c09a62e9f672418589e36f7395117fdb, user_id=e4ff09350e5040db91f9a60c6577f3e4, role_id=4621068972ba4f79bad104f956db04aa)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:44:38.183005 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e74ce1aa-f41d-4296-9323-1ddfe5ff2a55 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:44:38.262406 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 566/1123] 10.222.0.27 () {58 vars in 1439 bytes} [Tue Aug 30 01:44:38 2022] PUT /identity/v3/projects/c09a62e9f672418589e36f7395117fdb/users/e4ff09350e5040db91f9a60c6577f3e4/roles/4621068972ba4f79bad104f956db04aa => generated 0 bytes in 217 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 01:44:40.147245 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2ef1c6e-7641-4922-9f8d-af51f34dddef None None] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:44:40.147797 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2ef1c6e-7641-4922-9f8d-af51f34dddef None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:44:40.147797 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2ef1c6e-7641-4922-9f8d-af51f34dddef None None] PATH_INFO: `/` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:44:40.149094 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 558/1124] 10.222.0.27 () {54 vars in 927 bytes} [Tue Aug 30 01:44:40 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 01:44:40.154125 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c83b1f3b-825a-4894-86d2-d3d741f248e4 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:44:40.154522 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c83b1f3b-825a-4894-86d2-d3d741f248e4 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:44:40.154668 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c83b1f3b-825a-4894-86d2-d3d741f248e4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:44:40.172330 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-c83b1f3b-825a-4894-86d2-d3d741f248e4 None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 01:44:40.173522 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-c83b1f3b-825a-4894-86d2-d3d741f248e4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:44:40.238470 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 567/1125] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Aug 30 01:44:40 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 30 01:44:40.508025 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2ef1c6e-7641-4922-9f8d-af51f34dddef None None] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:44:40.606936 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2ef1c6e-7641-4922-9f8d-af51f34dddef None None] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:44:40.639579 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78827b42-6af4-4f0c-9639-b5b3fd4059e1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:44:40.640186 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78827b42-6af4-4f0c-9639-b5b3fd4059e1 None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:44:40.640464 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78827b42-6af4-4f0c-9639-b5b3fd4059e1 None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:44:40.640731 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78827b42-6af4-4f0c-9639-b5b3fd4059e1 None admin] PATH_INFO: `/v3/roles/4621068972ba4f79bad104f956db04aa` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:44:40.642134 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78827b42-6af4-4f0c-9639-b5b3fd4059e1 None admin] RBAC: Authorizing `identity:get_role(role_id=4621068972ba4f79bad104f956db04aa)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:44:40.643840 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78827b42-6af4-4f0c-9639-b5b3fd4059e1 None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:44:40.645122 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 559/1126] 10.222.0.27 () {56 vars in 1177 bytes} [Tue Aug 30 01:44:40 2022] GET /identity/v3/roles/4621068972ba4f79bad104f956db04aa => generated 223 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:44:40.650738 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c83b1f3b-825a-4894-86d2-d3d741f248e4 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:44:40.689613 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:44:40.748713 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c83b1f3b-825a-4894-86d2-d3d741f248e4 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:44:40.782032 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f194b5fc-2eca-4ba2-ae99-c80c750c58a1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:44:40.782584 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f194b5fc-2eca-4ba2-ae99-c80c750c58a1 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:44:40.782858 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f194b5fc-2eca-4ba2-ae99-c80c750c58a1 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:44:40.783125 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f194b5fc-2eca-4ba2-ae99-c80c750c58a1 None admin] PATH_INFO: `/v3/users/e4ff09350e5040db91f9a60c6577f3e4` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:44:40.784866 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f194b5fc-2eca-4ba2-ae99-c80c750c58a1 None admin] RBAC: Authorizing `identity:get_user(user_id=e4ff09350e5040db91f9a60c6577f3e4)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:44:40.786312 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f194b5fc-2eca-4ba2-ae99-c80c750c58a1 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:44:40.787883 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 568/1127] 10.222.0.27 () {56 vars in 1177 bytes} [Tue Aug 30 01:44:40 2022] GET /identity/v3/users/e4ff09350e5040db91f9a60c6577f3e4 => generated 286 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:44:40.793683 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78827b42-6af4-4f0c-9639-b5b3fd4059e1 None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:44:40.802660 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:44:40.891708 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78827b42-6af4-4f0c-9639-b5b3fd4059e1 None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:44:40.925385 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04a28768-15bb-4856-8111-600310bec745 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:44:40.926006 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04a28768-15bb-4856-8111-600310bec745 None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:44:40.926289 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04a28768-15bb-4856-8111-600310bec745 None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:44:40.926557 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04a28768-15bb-4856-8111-600310bec745 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:44:40.991790 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04a28768-15bb-4856-8111-600310bec745 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:44:40.993204 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04a28768-15bb-4856-8111-600310bec745 None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:44:41.000431 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:44:41.089080 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 560/1128] 10.222.0.27 () {58 vars in 1282 bytes} [Tue Aug 30 01:44:40 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 296 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Aug 30 01:44:41.095026 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f194b5fc-2eca-4ba2-ae99-c80c750c58a1 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:44:41.191484 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f194b5fc-2eca-4ba2-ae99-c80c750c58a1 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:44:41.198449 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:44:41.224835 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d4d3868-5c49-4ce4-a583-5b70d300e967 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:44:41.225442 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d4d3868-5c49-4ce4-a583-5b70d300e967 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:44:41.225750 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d4d3868-5c49-4ce4-a583-5b70d300e967 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:44:41.226020 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d4d3868-5c49-4ce4-a583-5b70d300e967 None admin] PATH_INFO: `/v3/projects/c09a62e9f672418589e36f7395117fdb` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:44:41.227370 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d4d3868-5c49-4ce4-a583-5b70d300e967 None admin] RBAC: Authorizing `identity:get_project(project_id=c09a62e9f672418589e36f7395117fdb)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:44:41.228806 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d4d3868-5c49-4ce4-a583-5b70d300e967 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:44:41.230481 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 569/1129] 10.222.0.27 () {56 vars in 1186 bytes} [Tue Aug 30 01:44:41 2022] GET /identity/v3/projects/c09a62e9f672418589e36f7395117fdb => generated 311 bytes in 136 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:44:41.238287 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04a28768-15bb-4856-8111-600310bec745 None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:44:41.332767 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04a28768-15bb-4856-8111-600310bec745 None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:44:41.339828 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:44:41.366542 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5daf8ddb-2da0-480c-b3bf-4eb38953afbe None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:44:41.367125 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5daf8ddb-2da0-480c-b3bf-4eb38953afbe None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:44:41.367404 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5daf8ddb-2da0-480c-b3bf-4eb38953afbe None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:44:41.367671 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5daf8ddb-2da0-480c-b3bf-4eb38953afbe None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:44:41.368389 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5daf8ddb-2da0-480c-b3bf-4eb38953afbe None admin] RBAC: Adding query filter params (role.id=4621068972ba4f79bad104f956db04aa, scope.project.id=c09a62e9f672418589e36f7395117fdb, user.id=e4ff09350e5040db91f9a60c6577f3e4) {{(pid=81578) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:44:41.368696 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5daf8ddb-2da0-480c-b3bf-4eb38953afbe None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:44:41.370138 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5daf8ddb-2da0-480c-b3bf-4eb38953afbe None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:44:41.375942 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 561/1130] 10.222.0.27 () {56 vars in 1374 bytes} [Tue Aug 30 01:44:41 2022] GET /identity/v3/role_assignments?user.id=e4ff09350e5040db91f9a60c6577f3e4&scope.project.id=c09a62e9f672418589e36f7395117fdb&role.id=4621068972ba4f79bad104f956db04aa => generated 608 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:44:43.261641 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8927915e-b2fb-4839-99c2-55bcf9d39d00 None None] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:44:43.262200 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8927915e-b2fb-4839-99c2-55bcf9d39d00 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:44:43.262200 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8927915e-b2fb-4839-99c2-55bcf9d39d00 None None] PATH_INFO: `/` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:44:43.263553 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 570/1131] 10.222.0.27 () {54 vars in 927 bytes} [Tue Aug 30 01:44:43 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 01:44:43.268597 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4ca0413-d8e2-4407-8fb5-641ee584a422 None None] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:44:43.269018 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4ca0413-d8e2-4407-8fb5-641ee584a422 None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:44:43.269127 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4ca0413-d8e2-4407-8fb5-641ee584a422 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:44:43.287311 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.auth.core [None req-b4ca0413-d8e2-4407-8fb5-641ee584a422 None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81578) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 01:44:43.288478 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-b4ca0413-d8e2-4407-8fb5-641ee584a422 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:44:43.353068 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 562/1132] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Aug 30 01:44:43 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Aug 30 01:44:43.629366 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8927915e-b2fb-4839-99c2-55bcf9d39d00 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:44:43.743051 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8927915e-b2fb-4839-99c2-55bcf9d39d00 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:44:43.782023 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9a098d0-e23b-42c1-add0-32a4afc91570 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:44:43.782650 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9a098d0-e23b-42c1-add0-32a4afc91570 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:44:43.782927 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9a098d0-e23b-42c1-add0-32a4afc91570 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:44:43.783197 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9a098d0-e23b-42c1-add0-32a4afc91570 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:44:43.784928 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9a098d0-e23b-42c1-add0-32a4afc91570 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:44:43.786448 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9a098d0-e23b-42c1-add0-32a4afc91570 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:44:43.788059 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 571/1133] 10.222.0.27 () {56 vars in 1108 bytes} [Tue Aug 30 01:44:43 2022] GET /identity/v3/domains/default => generated 199 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:44:43.794063 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4ca0413-d8e2-4407-8fb5-641ee584a422 None None] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:44:43.887928 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:44:43.890778 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4ca0413-d8e2-4407-8fb5-641ee584a422 None None] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:44:43.923915 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f542a5e0-1e1c-43db-ae6e-14018f8587a8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:44:43.924540 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f542a5e0-1e1c-43db-ae6e-14018f8587a8 None admin] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:44:43.924816 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f542a5e0-1e1c-43db-ae6e-14018f8587a8 None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:44:43.925082 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f542a5e0-1e1c-43db-ae6e-14018f8587a8 None admin] PATH_INFO: `/v3/projects` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:44:43.925750 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f542a5e0-1e1c-43db-ae6e-14018f8587a8 None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:44:43.927160 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f542a5e0-1e1c-43db-ae6e-14018f8587a8 None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:44:44.006291 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 563/1134] 10.222.0.27 () {60 vars in 1141 bytes} [Tue Aug 30 01:44:43 2022] POST /identity/v3/projects => generated 311 bytes in 213 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 01:44:45.948596 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d0341b4-788f-462e-b195-f663f2580baa None None] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:44:45.949138 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d0341b4-788f-462e-b195-f663f2580baa None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:44:45.949138 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d0341b4-788f-462e-b195-f663f2580baa None None] PATH_INFO: `/` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:44:45.950506 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 572/1135] 10.222.0.27 () {54 vars in 927 bytes} [Tue Aug 30 01:44:45 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 01:44:45.955723 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e1df040-d234-4bdb-b38d-55c7d87cb5ef None None] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:44:45.956153 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e1df040-d234-4bdb-b38d-55c7d87cb5ef None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:44:45.956260 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e1df040-d234-4bdb-b38d-55c7d87cb5ef None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:44:45.973962 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.auth.core [None req-9e1df040-d234-4bdb-b38d-55c7d87cb5ef None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81578) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 01:44:45.975066 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-9e1df040-d234-4bdb-b38d-55c7d87cb5ef None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:44:46.039649 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 564/1136] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Aug 30 01:44:45 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 30 01:44:46.310064 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d0341b4-788f-462e-b195-f663f2580baa None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:44:46.356489 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:44:46.414167 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d0341b4-788f-462e-b195-f663f2580baa None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:44:46.446713 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f209e34-bcd2-4d51-8db2-aa0f54e535aa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:44:46.447322 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f209e34-bcd2-4d51-8db2-aa0f54e535aa None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:44:46.447597 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f209e34-bcd2-4d51-8db2-aa0f54e535aa None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:44:46.447864 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f209e34-bcd2-4d51-8db2-aa0f54e535aa None admin] PATH_INFO: `/v3/domains/default` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:44:46.448952 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f209e34-bcd2-4d51-8db2-aa0f54e535aa None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:44:46.450399 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f209e34-bcd2-4d51-8db2-aa0f54e535aa None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:44:46.451945 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 573/1137] 10.222.0.27 () {56 vars in 1108 bytes} [Tue Aug 30 01:44:46 2022] GET /identity/v3/domains/default => generated 199 bytes in 142 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:44:46.458340 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e1df040-d234-4bdb-b38d-55c7d87cb5ef None None] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:44:46.535471 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:44:46.552504 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e1df040-d234-4bdb-b38d-55c7d87cb5ef None None] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:44:46.586005 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78e3aa88-d855-4085-b938-0f8195fdff79 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:44:46.586536 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78e3aa88-d855-4085-b938-0f8195fdff79 None admin] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:44:46.586813 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78e3aa88-d855-4085-b938-0f8195fdff79 None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:44:46.587084 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78e3aa88-d855-4085-b938-0f8195fdff79 None admin] PATH_INFO: `/v3/users` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:44:46.587737 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78e3aa88-d855-4085-b938-0f8195fdff79 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:44:46.589450 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78e3aa88-d855-4085-b938-0f8195fdff79 None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:44:46.673873 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 565/1138] 10.222.0.27 () {60 vars in 1132 bytes} [Tue Aug 30 01:44:46 2022] POST /identity/v3/users => generated 286 bytes in 216 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 01:44:48.608535 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a4c838e-8053-4c80-8f37-6cf9fd670502 None None] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:44:48.609101 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a4c838e-8053-4c80-8f37-6cf9fd670502 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:44:48.609101 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a4c838e-8053-4c80-8f37-6cf9fd670502 None None] PATH_INFO: `/` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:44:48.610449 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 574/1139] 10.222.0.27 () {54 vars in 927 bytes} [Tue Aug 30 01:44:48 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 01:44:48.615606 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bce925fa-9a6a-4b6d-a581-f5c2a9128409 None None] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:44:48.616024 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bce925fa-9a6a-4b6d-a581-f5c2a9128409 None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:44:48.616130 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bce925fa-9a6a-4b6d-a581-f5c2a9128409 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:44:48.634264 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.auth.core [None req-bce925fa-9a6a-4b6d-a581-f5c2a9128409 None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81578) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 01:44:48.635430 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-bce925fa-9a6a-4b6d-a581-f5c2a9128409 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:44:48.699555 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 566/1140] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Aug 30 01:44:48 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 30 01:44:48.967264 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a4c838e-8053-4c80-8f37-6cf9fd670502 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:44:49.066296 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a4c838e-8053-4c80-8f37-6cf9fd670502 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:44:49.076518 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:44:49.100028 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0828c321-217e-497c-bb93-42391aaa5368 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:44:49.100648 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0828c321-217e-497c-bb93-42391aaa5368 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:44:49.100924 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0828c321-217e-497c-bb93-42391aaa5368 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:44:49.101449 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0828c321-217e-497c-bb93-42391aaa5368 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:44:49.111753 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0828c321-217e-497c-bb93-42391aaa5368 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 30 01:44:49.116538 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application [None req-0828c321-217e-497c-bb93-42391aaa5368 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 30 01:44:49.116538 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 01:44:49.116538 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 01:44:49.116538 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 01:44:49.116538 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 01:44:49.116538 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 01:44:49.116538 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 01:44:49.116538 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 01:44:49.116538 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 01:44:49.116538 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 01:44:49.116538 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 01:44:49.116538 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 01:44:49.116538 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 30 01:44:49.116538 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 30 01:44:49.116538 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 30 01:44:49.116538 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 30 01:44:49.117806 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 30 01:44:49.117806 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 30 01:44:49.117806 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 30 01:44:49.117806 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 30 01:44:49.117806 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 01:44:49.117806 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 01:44:49.117806 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 01:44:49.117806 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 01:44:49.117806 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 01:44:49.117806 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 01:44:49.117806 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 01:44:49.117806 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application with Lock( Aug 30 01:44:49.117806 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 01:44:49.117806 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._enter() Aug 30 01:44:49.117806 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 01:44:49.117806 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 01:44:49.119059 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 01:44:49.119059 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.creator() Aug 30 01:44:49.119059 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 01:44:49.119059 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application created_value = creator( Aug 30 01:44:49.119059 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 30 01:44:49.119059 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 30 01:44:49.119059 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 30 01:44:49.119059 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 30 01:44:49.119059 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 30 01:44:49.119059 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 30 01:44:49.119059 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 30 01:44:49.119059 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Aug 30 01:44:49.119059 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 575/1141] 10.222.0.27 () {56 vars in 1096 bytes} [Tue Aug 30 01:44:48 2022] GET /identity/v3/roles/admin => generated 83 bytes in 151 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 01:44:49.123571 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bce925fa-9a6a-4b6d-a581-f5c2a9128409 None None] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:44:49.186169 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:44:49.217493 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bce925fa-9a6a-4b6d-a581-f5c2a9128409 None None] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:44:49.251240 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee34a154-0eee-48a2-b6db-d35dc190bef9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:44:49.251848 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee34a154-0eee-48a2-b6db-d35dc190bef9 None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:44:49.252672 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee34a154-0eee-48a2-b6db-d35dc190bef9 None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:44:49.252943 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee34a154-0eee-48a2-b6db-d35dc190bef9 None admin] PATH_INFO: `/v3/roles` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:44:49.253628 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee34a154-0eee-48a2-b6db-d35dc190bef9 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81578) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:44:49.253934 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee34a154-0eee-48a2-b6db-d35dc190bef9 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:44:49.255289 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee34a154-0eee-48a2-b6db-d35dc190bef9 None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:44:49.262998 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 567/1142] 10.222.0.27 () {56 vars in 1099 bytes} [Tue Aug 30 01:44:49 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:44:49.269267 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0828c321-217e-497c-bb93-42391aaa5368 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:44:49.366294 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0828c321-217e-497c-bb93-42391aaa5368 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:44:49.399174 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77734487-6bd6-425c-b36a-b477e4d45d9f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:44:49.399782 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77734487-6bd6-425c-b36a-b477e4d45d9f None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:44:49.400058 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77734487-6bd6-425c-b36a-b477e4d45d9f None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:44:49.400326 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77734487-6bd6-425c-b36a-b477e4d45d9f None admin] PATH_INFO: `/v3/users/8bc58fd0beca43dc9ec9900b1aec5118` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:44:49.413032 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77734487-6bd6-425c-b36a-b477e4d45d9f None admin] RBAC: Authorizing `identity:get_user(user_id=8bc58fd0beca43dc9ec9900b1aec5118)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:44:49.414532 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77734487-6bd6-425c-b36a-b477e4d45d9f None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:44:49.416097 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 576/1143] 10.222.0.27 () {56 vars in 1177 bytes} [Tue Aug 30 01:44:49 2022] GET /identity/v3/users/8bc58fd0beca43dc9ec9900b1aec5118 => generated 286 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 30 01:44:49.422009 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee34a154-0eee-48a2-b6db-d35dc190bef9 None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:44:49.521040 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee34a154-0eee-48a2-b6db-d35dc190bef9 None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:44:49.547464 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:44:49.553061 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9af4740f-a11b-44ca-8ed0-3bfb1f6b92d3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:44:49.553674 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9af4740f-a11b-44ca-8ed0-3bfb1f6b92d3 None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:44:49.553951 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9af4740f-a11b-44ca-8ed0-3bfb1f6b92d3 None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:44:49.554215 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9af4740f-a11b-44ca-8ed0-3bfb1f6b92d3 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:44:49.619928 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9af4740f-a11b-44ca-8ed0-3bfb1f6b92d3 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:44:49.621353 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9af4740f-a11b-44ca-8ed0-3bfb1f6b92d3 None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:44:49.716964 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 568/1144] 10.222.0.27 () {58 vars in 1282 bytes} [Tue Aug 30 01:44:49 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 295 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Aug 30 01:44:49.722656 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77734487-6bd6-425c-b36a-b477e4d45d9f None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:44:49.750428 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:44:49.818680 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77734487-6bd6-425c-b36a-b477e4d45d9f None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:44:49.850677 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30fa4787-75b9-46af-9b10-6561146b5e5a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:44:49.851591 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30fa4787-75b9-46af-9b10-6561146b5e5a None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:44:49.851866 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30fa4787-75b9-46af-9b10-6561146b5e5a None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:44:49.852131 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30fa4787-75b9-46af-9b10-6561146b5e5a None admin] PATH_INFO: `/v3/projects/1067e44ff54d4865be14149d2ec5dc49` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:44:49.853219 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30fa4787-75b9-46af-9b10-6561146b5e5a None admin] RBAC: Authorizing `identity:get_project(project_id=1067e44ff54d4865be14149d2ec5dc49)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:44:49.854669 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30fa4787-75b9-46af-9b10-6561146b5e5a None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:44:49.856266 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 577/1145] 10.222.0.27 () {56 vars in 1186 bytes} [Tue Aug 30 01:44:49 2022] GET /identity/v3/projects/1067e44ff54d4865be14149d2ec5dc49 => generated 311 bytes in 134 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:44:49.862920 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9af4740f-a11b-44ca-8ed0-3bfb1f6b92d3 None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:44:49.892201 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:44:49.959910 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9af4740f-a11b-44ca-8ed0-3bfb1f6b92d3 None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:44:49.992957 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ba79c82-8814-47ba-9914-4e5fef1e244d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:44:49.993583 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ba79c82-8814-47ba-9914-4e5fef1e244d None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:44:49.993864 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ba79c82-8814-47ba-9914-4e5fef1e244d None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:44:49.994134 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ba79c82-8814-47ba-9914-4e5fef1e244d None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:44:49.994850 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ba79c82-8814-47ba-9914-4e5fef1e244d None admin] RBAC: Adding query filter params (role.id=ead5fac86a0747ea8823a82efcea8ed7, scope.project.id=1067e44ff54d4865be14149d2ec5dc49, user.id=8bc58fd0beca43dc9ec9900b1aec5118) {{(pid=81578) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:44:49.995157 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ba79c82-8814-47ba-9914-4e5fef1e244d None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:44:49.996566 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ba79c82-8814-47ba-9914-4e5fef1e244d None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:44:50.001739 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 569/1146] 10.222.0.27 () {56 vars in 1374 bytes} [Tue Aug 30 01:44:49 2022] GET /identity/v3/role_assignments?user.id=8bc58fd0beca43dc9ec9900b1aec5118&scope.project.id=1067e44ff54d4865be14149d2ec5dc49&role.id=ead5fac86a0747ea8823a82efcea8ed7 => generated 259 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 30 01:44:51.850187 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76930375-815f-4d1d-b9c8-5aec33674355 None None] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:44:51.850727 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76930375-815f-4d1d-b9c8-5aec33674355 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:44:51.850727 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76930375-815f-4d1d-b9c8-5aec33674355 None None] PATH_INFO: `/` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:44:51.852322 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 578/1147] 10.222.0.27 () {54 vars in 927 bytes} [Tue Aug 30 01:44:51 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 01:44:51.857279 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aefcd366-189e-402c-a2f4-10497c1130c6 None None] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:44:51.857733 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aefcd366-189e-402c-a2f4-10497c1130c6 None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:44:51.857826 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aefcd366-189e-402c-a2f4-10497c1130c6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:44:51.875961 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.auth.core [None req-aefcd366-189e-402c-a2f4-10497c1130c6 None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81578) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 01:44:51.877170 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-aefcd366-189e-402c-a2f4-10497c1130c6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:44:51.942447 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 570/1148] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Aug 30 01:44:51 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 30 01:44:52.207227 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76930375-815f-4d1d-b9c8-5aec33674355 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:44:52.299800 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:44:52.306306 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76930375-815f-4d1d-b9c8-5aec33674355 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:44:52.340789 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-beecde11-f3ac-4724-8c01-98f7eeea3887 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:44:52.341444 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-beecde11-f3ac-4724-8c01-98f7eeea3887 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:44:52.341946 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-beecde11-f3ac-4724-8c01-98f7eeea3887 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:44:52.342436 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-beecde11-f3ac-4724-8c01-98f7eeea3887 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:44:52.352984 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-beecde11-f3ac-4724-8c01-98f7eeea3887 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 30 01:44:52.357735 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application [None req-beecde11-f3ac-4724-8c01-98f7eeea3887 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 30 01:44:52.357735 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 01:44:52.357735 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 01:44:52.357735 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 01:44:52.357735 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 01:44:52.357735 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 01:44:52.357735 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 01:44:52.357735 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 01:44:52.357735 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 01:44:52.357735 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 01:44:52.357735 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 01:44:52.357735 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 01:44:52.357735 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 30 01:44:52.357735 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 30 01:44:52.357735 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 30 01:44:52.357735 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 30 01:44:52.358841 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 30 01:44:52.358841 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 30 01:44:52.358841 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 30 01:44:52.358841 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 30 01:44:52.358841 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 01:44:52.358841 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 01:44:52.358841 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 01:44:52.358841 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 01:44:52.358841 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 01:44:52.358841 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 01:44:52.358841 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 01:44:52.358841 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application with Lock( Aug 30 01:44:52.358841 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 01:44:52.358841 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._enter() Aug 30 01:44:52.358841 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 01:44:52.358841 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 01:44:52.359927 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 01:44:52.359927 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.creator() Aug 30 01:44:52.359927 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 01:44:52.359927 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application created_value = creator( Aug 30 01:44:52.359927 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 30 01:44:52.359927 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 30 01:44:52.359927 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 30 01:44:52.359927 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 30 01:44:52.359927 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 30 01:44:52.359927 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 30 01:44:52.359927 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 30 01:44:52.359927 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Aug 30 01:44:52.359927 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 579/1149] 10.222.0.27 () {56 vars in 1096 bytes} [Tue Aug 30 01:44:52 2022] GET /identity/v3/roles/admin => generated 83 bytes in 152 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 01:44:52.364991 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aefcd366-189e-402c-a2f4-10497c1130c6 None None] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:44:52.389438 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:44:52.462814 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aefcd366-189e-402c-a2f4-10497c1130c6 None None] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:44:52.495522 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a58b4b7b-809a-4ac6-bffa-a7f6b6482edf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:44:52.496116 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a58b4b7b-809a-4ac6-bffa-a7f6b6482edf None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:44:52.496393 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a58b4b7b-809a-4ac6-bffa-a7f6b6482edf None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:44:52.496659 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a58b4b7b-809a-4ac6-bffa-a7f6b6482edf None admin] PATH_INFO: `/v3/roles` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:44:52.497312 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a58b4b7b-809a-4ac6-bffa-a7f6b6482edf None admin] RBAC: Adding query filter params (name=admin) {{(pid=81578) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:44:52.497648 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a58b4b7b-809a-4ac6-bffa-a7f6b6482edf None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:44:52.498998 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a58b4b7b-809a-4ac6-bffa-a7f6b6482edf None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:44:52.507363 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 571/1150] 10.222.0.27 () {56 vars in 1099 bytes} [Tue Aug 30 01:44:52 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:44:52.513140 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-beecde11-f3ac-4724-8c01-98f7eeea3887 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:44:52.619572 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-beecde11-f3ac-4724-8c01-98f7eeea3887 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:44:52.652156 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acb4a6ff-f86a-4bd6-8eeb-66b55d71eff6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:44:52.652780 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acb4a6ff-f86a-4bd6-8eeb-66b55d71eff6 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:44:52.653056 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acb4a6ff-f86a-4bd6-8eeb-66b55d71eff6 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:44:52.653327 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acb4a6ff-f86a-4bd6-8eeb-66b55d71eff6 None admin] PATH_INFO: `/v3/users/8bc58fd0beca43dc9ec9900b1aec5118` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:44:52.654522 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-acb4a6ff-f86a-4bd6-8eeb-66b55d71eff6 None admin] RBAC: Authorizing `identity:get_user(user_id=8bc58fd0beca43dc9ec9900b1aec5118)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:44:52.655947 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-acb4a6ff-f86a-4bd6-8eeb-66b55d71eff6 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:44:52.657537 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 580/1151] 10.222.0.27 () {56 vars in 1177 bytes} [Tue Aug 30 01:44:52 2022] GET /identity/v3/users/8bc58fd0beca43dc9ec9900b1aec5118 => generated 286 bytes in 145 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:44:52.664009 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a58b4b7b-809a-4ac6-bffa-a7f6b6482edf None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:44:52.754927 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:44:52.769867 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a58b4b7b-809a-4ac6-bffa-a7f6b6482edf None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:44:52.803216 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0030743-5467-4751-8d8c-5ee9d177b198 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:44:52.803840 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0030743-5467-4751-8d8c-5ee9d177b198 None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:44:52.804119 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0030743-5467-4751-8d8c-5ee9d177b198 None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:44:52.804390 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0030743-5467-4751-8d8c-5ee9d177b198 None admin] PATH_INFO: `/v3/projects/1067e44ff54d4865be14149d2ec5dc49` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:44:52.805506 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0030743-5467-4751-8d8c-5ee9d177b198 None admin] RBAC: Authorizing `identity:get_project(project_id=1067e44ff54d4865be14149d2ec5dc49)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:44:52.806926 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0030743-5467-4751-8d8c-5ee9d177b198 None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:44:52.808533 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 572/1152] 10.222.0.27 () {56 vars in 1186 bytes} [Tue Aug 30 01:44:52 2022] GET /identity/v3/projects/1067e44ff54d4865be14149d2ec5dc49 => generated 311 bytes in 145 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:44:52.814529 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acb4a6ff-f86a-4bd6-8eeb-66b55d71eff6 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:44:52.839986 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:44:52.913364 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acb4a6ff-f86a-4bd6-8eeb-66b55d71eff6 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:44:52.945287 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b12cc78d-5916-45a7-9e77-96da768fa2af None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:44:52.945956 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b12cc78d-5916-45a7-9e77-96da768fa2af None admin] REQUEST_METHOD: `PUT` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:44:52.946235 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b12cc78d-5916-45a7-9e77-96da768fa2af None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:44:52.946502 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b12cc78d-5916-45a7-9e77-96da768fa2af None admin] PATH_INFO: `/v3/projects/1067e44ff54d4865be14149d2ec5dc49/users/8bc58fd0beca43dc9ec9900b1aec5118/roles/ead5fac86a0747ea8823a82efcea8ed7` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:44:52.948374 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b12cc78d-5916-45a7-9e77-96da768fa2af None admin] RBAC: Authorizing `identity:create_grant(project_id=1067e44ff54d4865be14149d2ec5dc49, user_id=8bc58fd0beca43dc9ec9900b1aec5118, role_id=ead5fac86a0747ea8823a82efcea8ed7)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:44:52.949818 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b12cc78d-5916-45a7-9e77-96da768fa2af None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:44:53.015937 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 581/1153] 10.222.0.27 () {58 vars in 1439 bytes} [Tue Aug 30 01:44:52 2022] PUT /identity/v3/projects/1067e44ff54d4865be14149d2ec5dc49/users/8bc58fd0beca43dc9ec9900b1aec5118/roles/ead5fac86a0747ea8823a82efcea8ed7 => generated 0 bytes in 202 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 01:44:54.899276 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0014e5dc-cfc1-4c68-8f20-317a1ff9ed1c None None] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:44:54.899902 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0014e5dc-cfc1-4c68-8f20-317a1ff9ed1c None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:44:54.899902 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0014e5dc-cfc1-4c68-8f20-317a1ff9ed1c None None] PATH_INFO: `/` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:44:54.901143 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 573/1154] 10.222.0.27 () {54 vars in 927 bytes} [Tue Aug 30 01:44:54 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 01:44:54.906461 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc2bff6e-4d67-46b8-99d8-55445ce9d5f9 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:44:54.906857 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc2bff6e-4d67-46b8-99d8-55445ce9d5f9 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:44:54.906997 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc2bff6e-4d67-46b8-99d8-55445ce9d5f9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:44:54.925658 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-dc2bff6e-4d67-46b8-99d8-55445ce9d5f9 None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 01:44:54.926822 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-dc2bff6e-4d67-46b8-99d8-55445ce9d5f9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:44:54.991284 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 582/1155] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Aug 30 01:44:54 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 30 01:44:55.259240 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0014e5dc-cfc1-4c68-8f20-317a1ff9ed1c None None] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:44:55.360094 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0014e5dc-cfc1-4c68-8f20-317a1ff9ed1c None None] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:44:55.394743 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3755e1ea-e6dd-4bd2-ae5a-11b5f4e7f8ea None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:44:55.395359 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3755e1ea-e6dd-4bd2-ae5a-11b5f4e7f8ea None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:44:55.395635 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3755e1ea-e6dd-4bd2-ae5a-11b5f4e7f8ea None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:44:55.395901 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3755e1ea-e6dd-4bd2-ae5a-11b5f4e7f8ea None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:44:55.405936 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3755e1ea-e6dd-4bd2-ae5a-11b5f4e7f8ea None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 30 01:44:55.410689 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application [None req-3755e1ea-e6dd-4bd2-ae5a-11b5f4e7f8ea None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 30 01:44:55.410689 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 01:44:55.410689 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 01:44:55.410689 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 01:44:55.410689 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 01:44:55.410689 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 01:44:55.410689 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 01:44:55.410689 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 01:44:55.410689 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 01:44:55.410689 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 01:44:55.410689 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 01:44:55.410689 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 01:44:55.410689 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 30 01:44:55.410689 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 30 01:44:55.410689 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 30 01:44:55.410689 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 30 01:44:55.411777 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 30 01:44:55.411777 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 30 01:44:55.411777 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 30 01:44:55.411777 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 30 01:44:55.411777 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 01:44:55.411777 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 01:44:55.411777 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 01:44:55.411777 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 01:44:55.411777 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 01:44:55.411777 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 01:44:55.411777 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 01:44:55.411777 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application with Lock( Aug 30 01:44:55.411777 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 01:44:55.411777 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._enter() Aug 30 01:44:55.411777 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 01:44:55.411777 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 01:44:55.412819 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 01:44:55.412819 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.creator() Aug 30 01:44:55.412819 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 01:44:55.412819 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application created_value = creator( Aug 30 01:44:55.412819 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 30 01:44:55.412819 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 30 01:44:55.412819 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 30 01:44:55.412819 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 30 01:44:55.412819 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 30 01:44:55.412819 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 30 01:44:55.412819 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 30 01:44:55.412819 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application Aug 30 01:44:55.412819 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 574/1156] 10.222.0.27 () {56 vars in 1096 bytes} [Tue Aug 30 01:44:55 2022] GET /identity/v3/roles/admin => generated 83 bytes in 153 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 01:44:55.417960 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc2bff6e-4d67-46b8-99d8-55445ce9d5f9 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:44:55.432589 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:44:55.514643 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc2bff6e-4d67-46b8-99d8-55445ce9d5f9 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:44:55.547433 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae81ca44-64e0-41fc-a995-ed8d6050dc1b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:44:55.548032 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae81ca44-64e0-41fc-a995-ed8d6050dc1b None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:44:55.548312 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae81ca44-64e0-41fc-a995-ed8d6050dc1b None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:44:55.548611 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae81ca44-64e0-41fc-a995-ed8d6050dc1b None admin] PATH_INFO: `/v3/roles` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:44:55.549269 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae81ca44-64e0-41fc-a995-ed8d6050dc1b None admin] RBAC: Adding query filter params (name=admin) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:44:55.549601 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae81ca44-64e0-41fc-a995-ed8d6050dc1b None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:44:55.550972 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae81ca44-64e0-41fc-a995-ed8d6050dc1b None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:44:55.558731 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 583/1157] 10.222.0.27 () {56 vars in 1099 bytes} [Tue Aug 30 01:44:55 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 141 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:44:55.565203 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3755e1ea-e6dd-4bd2-ae5a-11b5f4e7f8ea None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:44:55.580849 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:44:55.665217 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3755e1ea-e6dd-4bd2-ae5a-11b5f4e7f8ea None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:44:55.698318 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1177042-9084-4c36-8c1d-0edf8d2cd932 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:44:55.698929 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1177042-9084-4c36-8c1d-0edf8d2cd932 None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:44:55.699200 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1177042-9084-4c36-8c1d-0edf8d2cd932 None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:44:55.699465 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1177042-9084-4c36-8c1d-0edf8d2cd932 None admin] PATH_INFO: `/v3/users/8bc58fd0beca43dc9ec9900b1aec5118` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:44:55.700556 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1177042-9084-4c36-8c1d-0edf8d2cd932 None admin] RBAC: Authorizing `identity:get_user(user_id=8bc58fd0beca43dc9ec9900b1aec5118)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:44:55.702001 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1177042-9084-4c36-8c1d-0edf8d2cd932 None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:44:55.703559 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 575/1158] 10.222.0.27 () {56 vars in 1177 bytes} [Tue Aug 30 01:44:55 2022] GET /identity/v3/users/8bc58fd0beca43dc9ec9900b1aec5118 => generated 286 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:44:55.710114 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae81ca44-64e0-41fc-a995-ed8d6050dc1b None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:44:55.787392 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:44:55.809251 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae81ca44-64e0-41fc-a995-ed8d6050dc1b None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:44:55.842087 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7ebff22-ef71-4f98-8b5c-44bcee85c16b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:44:55.842684 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7ebff22-ef71-4f98-8b5c-44bcee85c16b None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:44:55.842959 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7ebff22-ef71-4f98-8b5c-44bcee85c16b None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:44:55.843224 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7ebff22-ef71-4f98-8b5c-44bcee85c16b None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:44:55.907496 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7ebff22-ef71-4f98-8b5c-44bcee85c16b None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:44:55.909610 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7ebff22-ef71-4f98-8b5c-44bcee85c16b None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:44:55.980919 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:44:56.003175 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 584/1159] 10.222.0.27 () {58 vars in 1282 bytes} [Tue Aug 30 01:44:55 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 294 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Aug 30 01:44:56.009117 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1177042-9084-4c36-8c1d-0edf8d2cd932 None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:44:56.087843 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:44:56.105570 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1177042-9084-4c36-8c1d-0edf8d2cd932 None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:44:56.138376 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2e39c70-757f-4d13-827b-b0a6f806102c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:44:56.138987 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2e39c70-757f-4d13-827b-b0a6f806102c None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:44:56.139261 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2e39c70-757f-4d13-827b-b0a6f806102c None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:44:56.139532 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2e39c70-757f-4d13-827b-b0a6f806102c None admin] PATH_INFO: `/v3/projects/1067e44ff54d4865be14149d2ec5dc49` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:44:56.140613 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2e39c70-757f-4d13-827b-b0a6f806102c None admin] RBAC: Authorizing `identity:get_project(project_id=1067e44ff54d4865be14149d2ec5dc49)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:44:56.142049 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2e39c70-757f-4d13-827b-b0a6f806102c None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:44:56.144265 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 576/1160] 10.222.0.27 () {56 vars in 1186 bytes} [Tue Aug 30 01:44:56 2022] GET /identity/v3/projects/1067e44ff54d4865be14149d2ec5dc49 => generated 311 bytes in 136 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:44:56.150460 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7ebff22-ef71-4f98-8b5c-44bcee85c16b None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:44:56.246616 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7ebff22-ef71-4f98-8b5c-44bcee85c16b None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:44:56.279681 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94a88f5c-0cf7-482e-9af0-f2fd26c212fb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:44:56.280261 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94a88f5c-0cf7-482e-9af0-f2fd26c212fb None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:44:56.280535 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94a88f5c-0cf7-482e-9af0-f2fd26c212fb None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:44:56.280799 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94a88f5c-0cf7-482e-9af0-f2fd26c212fb None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:44:56.281532 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94a88f5c-0cf7-482e-9af0-f2fd26c212fb None admin] RBAC: Adding query filter params (role.id=ead5fac86a0747ea8823a82efcea8ed7, scope.project.id=1067e44ff54d4865be14149d2ec5dc49, user.id=8bc58fd0beca43dc9ec9900b1aec5118) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:44:56.281834 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94a88f5c-0cf7-482e-9af0-f2fd26c212fb None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:44:56.283809 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94a88f5c-0cf7-482e-9af0-f2fd26c212fb None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:44:56.289296 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 585/1161] 10.222.0.27 () {56 vars in 1374 bytes} [Tue Aug 30 01:44:56 2022] GET /identity/v3/role_assignments?user.id=8bc58fd0beca43dc9ec9900b1aec5118&scope.project.id=1067e44ff54d4865be14149d2ec5dc49&role.id=ead5fac86a0747ea8823a82efcea8ed7 => generated 608 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 30 01:44:58.207029 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c24380c-77e6-4f1b-b9a8-ba0944c664b0 None None] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:44:58.207544 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c24380c-77e6-4f1b-b9a8-ba0944c664b0 None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:44:58.207544 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c24380c-77e6-4f1b-b9a8-ba0944c664b0 None None] PATH_INFO: `/` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:44:58.208911 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 577/1162] 10.222.0.27 () {54 vars in 927 bytes} [Tue Aug 30 01:44:58 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 01:44:58.213959 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85368ffe-58ce-4c0e-885c-0773620b8e62 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:44:58.214376 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85368ffe-58ce-4c0e-885c-0773620b8e62 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:44:58.214501 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85368ffe-58ce-4c0e-885c-0773620b8e62 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:44:58.233368 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-85368ffe-58ce-4c0e-885c-0773620b8e62 None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 01:44:58.234578 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-85368ffe-58ce-4c0e-885c-0773620b8e62 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:44:58.268070 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:44:58.299371 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 586/1163] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Aug 30 01:44:58 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 30 01:44:58.566216 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c24380c-77e6-4f1b-b9a8-ba0944c664b0 None None] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:44:58.664867 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c24380c-77e6-4f1b-b9a8-ba0944c664b0 None None] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:44:58.697059 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-318e9e50-0be2-4dc0-a23c-cafc7bea301a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:44:58.697673 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-318e9e50-0be2-4dc0-a23c-cafc7bea301a None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:44:58.697948 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-318e9e50-0be2-4dc0-a23c-cafc7bea301a None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:44:58.698214 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-318e9e50-0be2-4dc0-a23c-cafc7bea301a None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:44:58.716672 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:44:58.763697 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-318e9e50-0be2-4dc0-a23c-cafc7bea301a None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:44:58.765121 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-318e9e50-0be2-4dc0-a23c-cafc7bea301a None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:44:58.875458 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 578/1164] 10.222.0.27 () {58 vars in 1282 bytes} [Tue Aug 30 01:44:58 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 310 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Aug 30 01:44:58.881545 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85368ffe-58ce-4c0e-885c-0773620b8e62 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:44:58.977192 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85368ffe-58ce-4c0e-885c-0773620b8e62 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:44:59.008996 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e393803-c630-4d56-a5e9-405072ae0a5f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:44:59.009642 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e393803-c630-4d56-a5e9-405072ae0a5f None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:44:59.009919 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e393803-c630-4d56-a5e9-405072ae0a5f None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:44:59.010181 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e393803-c630-4d56-a5e9-405072ae0a5f None admin] PATH_INFO: `/v3/domains/swift_test` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:44:59.016514 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e393803-c630-4d56-a5e9-405072ae0a5f None admin] RBAC: Authorizing `identity:get_domain(domain_id=swift_test)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:44:59.018536 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e393803-c630-4d56-a5e9-405072ae0a5f None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:44:59.029419 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application [None req-9e393803-c630-4d56-a5e9-405072ae0a5f None admin] Could not find domain: swift_test.: keystone.exception.DomainNotFound: Could not find domain: swift_test. Aug 30 01:44:59.029419 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 01:44:59.029419 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 712, in get_domain Aug 30 01:44:59.029419 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application project = self.driver.get_project(domain_id) Aug 30 01:44:59.029419 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 30 01:44:59.029419 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 30 01:44:59.029419 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 30 01:44:59.029419 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 30 01:44:59.029419 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: swift_test. Aug 30 01:44:59.029419 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Aug 30 01:44:59.029419 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 30 01:44:59.029419 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Aug 30 01:44:59.029419 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 01:44:59.029419 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 01:44:59.029419 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 01:44:59.029419 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 01:44:59.029419 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 01:44:59.030657 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 01:44:59.030657 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 01:44:59.030657 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 01:44:59.030657 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 01:44:59.030657 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 01:44:59.030657 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 01:44:59.030657 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 30 01:44:59.030657 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 30 01:44:59.030657 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 30 01:44:59.030657 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 30 01:44:59.030657 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 01:44:59.030657 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 01:44:59.030657 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 01:44:59.030657 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 01:44:59.030657 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 01:44:59.030657 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 01:44:59.031701 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 01:44:59.031701 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application with Lock( Aug 30 01:44:59.031701 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 01:44:59.031701 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._enter() Aug 30 01:44:59.031701 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 01:44:59.031701 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 01:44:59.031701 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 01:44:59.031701 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.creator() Aug 30 01:44:59.031701 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 01:44:59.031701 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application created_value = creator( Aug 30 01:44:59.031701 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 718, in get_domain Aug 30 01:44:59.031701 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 30 01:44:59.031701 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: swift_test. Aug 30 01:44:59.031701 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Aug 30 01:44:59.031701 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 587/1165] 10.222.0.27 () {56 vars in 1117 bytes} [Tue Aug 30 01:44:58 2022] GET /identity/v3/domains/swift_test => generated 90 bytes in 150 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 01:44:59.036983 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-318e9e50-0be2-4dc0-a23c-cafc7bea301a None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:44:59.084270 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:44:59.132670 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-318e9e50-0be2-4dc0-a23c-cafc7bea301a None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:44:59.178145 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eac826aa-ffef-4f39-b36a-ebb59e076e25 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:44:59.178691 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eac826aa-ffef-4f39-b36a-ebb59e076e25 None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:44:59.178966 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eac826aa-ffef-4f39-b36a-ebb59e076e25 None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:44:59.179233 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eac826aa-ffef-4f39-b36a-ebb59e076e25 None admin] PATH_INFO: `/v3/domains` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:44:59.179867 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eac826aa-ffef-4f39-b36a-ebb59e076e25 None admin] RBAC: Adding query filter params (name=swift_test) {{(pid=81578) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:44:59.180169 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eac826aa-ffef-4f39-b36a-ebb59e076e25 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:44:59.181647 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eac826aa-ffef-4f39-b36a-ebb59e076e25 None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:44:59.190408 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 579/1166] 10.222.0.27 () {56 vars in 1115 bytes} [Tue Aug 30 01:44:59 2022] GET /identity/v3/domains?name=swift_test => generated 125 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:45:01.078395 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a9d0279-9751-47ff-bd96-6d10006b03f2 None None] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:45:01.078909 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a9d0279-9751-47ff-bd96-6d10006b03f2 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:45:01.078909 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a9d0279-9751-47ff-bd96-6d10006b03f2 None None] PATH_INFO: `/` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:45:01.080785 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 588/1167] 10.222.0.27 () {54 vars in 927 bytes} [Tue Aug 30 01:45:01 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 01:45:01.085831 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db73fa40-e680-46ac-b7be-ca4b963788e5 None None] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:45:01.086259 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db73fa40-e680-46ac-b7be-ca4b963788e5 None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:45:01.086368 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db73fa40-e680-46ac-b7be-ca4b963788e5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:45:01.104732 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.auth.core [None req-db73fa40-e680-46ac-b7be-ca4b963788e5 None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81578) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 01:45:01.105924 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-db73fa40-e680-46ac-b7be-ca4b963788e5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:45:01.178545 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 580/1168] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Aug 30 01:45:01 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 30 01:45:01.452005 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a9d0279-9751-47ff-bd96-6d10006b03f2 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:45:01.484315 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:45:01.555196 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a9d0279-9751-47ff-bd96-6d10006b03f2 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:45:01.587248 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6c4c9c9-33a6-4eed-904d-dc56f99512c0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:45:01.587847 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6c4c9c9-33a6-4eed-904d-dc56f99512c0 None admin] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:45:01.588123 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6c4c9c9-33a6-4eed-904d-dc56f99512c0 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:45:01.588393 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6c4c9c9-33a6-4eed-904d-dc56f99512c0 None admin] PATH_INFO: `/v3/domains` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:45:01.589061 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6c4c9c9-33a6-4eed-904d-dc56f99512c0 None admin] RBAC: Authorizing `identity:create_domain()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:45:01.590455 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6c4c9c9-33a6-4eed-904d-dc56f99512c0 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:45:01.686230 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 589/1169] 10.222.0.27 () {60 vars in 1138 bytes} [Tue Aug 30 01:45:01 2022] POST /identity/v3/domains => generated 267 bytes in 235 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 01:45:03.623530 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a873e7e7-a1d5-4e38-8b4c-474dbe702887 None None] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:45:03.624093 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a873e7e7-a1d5-4e38-8b4c-474dbe702887 None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:45:03.624093 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a873e7e7-a1d5-4e38-8b4c-474dbe702887 None None] PATH_INFO: `/` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:45:03.625391 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 581/1170] 10.222.0.27 () {54 vars in 927 bytes} [Tue Aug 30 01:45:03 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 01:45:03.631198 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3053a3b-7733-4f26-846a-5bd6f64402ca None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:45:03.631631 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3053a3b-7733-4f26-846a-5bd6f64402ca None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:45:03.631734 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3053a3b-7733-4f26-846a-5bd6f64402ca None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:45:03.650141 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-e3053a3b-7733-4f26-846a-5bd6f64402ca None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 01:45:03.651317 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-e3053a3b-7733-4f26-846a-5bd6f64402ca None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:45:03.719844 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 590/1171] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Aug 30 01:45:03 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 30 01:45:03.989858 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a873e7e7-a1d5-4e38-8b4c-474dbe702887 None None] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:45:04.035696 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:45:04.088489 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a873e7e7-a1d5-4e38-8b4c-474dbe702887 None None] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:45:04.121343 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d085bcb-b018-4d24-be70-a2738e23bb73 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:45:04.121979 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d085bcb-b018-4d24-be70-a2738e23bb73 None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:45:04.122258 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d085bcb-b018-4d24-be70-a2738e23bb73 None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:45:04.122527 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d085bcb-b018-4d24-be70-a2738e23bb73 None admin] PATH_INFO: `/v3/domains/b111a0b8935c40a1b0197f851c0c778f` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:45:04.132623 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d085bcb-b018-4d24-be70-a2738e23bb73 None admin] RBAC: Authorizing `identity:get_domain(domain_id=b111a0b8935c40a1b0197f851c0c778f)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:45:04.134096 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d085bcb-b018-4d24-be70-a2738e23bb73 None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:45:04.135661 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 582/1172] 10.222.0.27 () {56 vars in 1183 bytes} [Tue Aug 30 01:45:03 2022] GET /identity/v3/domains/b111a0b8935c40a1b0197f851c0c778f => generated 267 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:45:04.141538 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3053a3b-7733-4f26-846a-5bd6f64402ca None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:45:04.163128 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:45:04.239097 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3053a3b-7733-4f26-846a-5bd6f64402ca None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:45:04.272978 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ed617f2-0853-425e-8bf1-5ebee0235bc9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:45:04.273532 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ed617f2-0853-425e-8bf1-5ebee0235bc9 None admin] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:45:04.273815 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ed617f2-0853-425e-8bf1-5ebee0235bc9 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:45:04.274085 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ed617f2-0853-425e-8bf1-5ebee0235bc9 None admin] PATH_INFO: `/v3/projects` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:45:04.274721 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ed617f2-0853-425e-8bf1-5ebee0235bc9 None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:45:04.276137 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ed617f2-0853-425e-8bf1-5ebee0235bc9 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:45:04.359279 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 591/1173] 10.222.0.27 () {60 vars in 1141 bytes} [Tue Aug 30 01:45:04 2022] POST /identity/v3/projects => generated 361 bytes in 218 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 01:45:06.269801 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a64f5da6-dd18-457f-9ed5-8536e72a92ac None None] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:45:06.270382 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a64f5da6-dd18-457f-9ed5-8536e72a92ac None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:45:06.270382 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a64f5da6-dd18-457f-9ed5-8536e72a92ac None None] PATH_INFO: `/` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:45:06.271668 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 583/1174] 10.222.0.27 () {54 vars in 927 bytes} [Tue Aug 30 01:45:06 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 01:45:06.276691 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93df6c87-2b48-4cd1-a5f4-3872546455e9 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:45:06.277127 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93df6c87-2b48-4cd1-a5f4-3872546455e9 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:45:06.277226 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93df6c87-2b48-4cd1-a5f4-3872546455e9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:45:06.295732 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-93df6c87-2b48-4cd1-a5f4-3872546455e9 None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 01:45:06.296917 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-93df6c87-2b48-4cd1-a5f4-3872546455e9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:45:06.361832 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 592/1175] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Aug 30 01:45:06 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 30 01:45:06.631936 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a64f5da6-dd18-457f-9ed5-8536e72a92ac None None] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:45:06.732612 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a64f5da6-dd18-457f-9ed5-8536e72a92ac None None] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:45:06.734979 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:45:06.766136 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72d0dee5-5c72-4d20-aa0b-ce70c40d1d85 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:45:06.767499 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72d0dee5-5c72-4d20-aa0b-ce70c40d1d85 None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:45:06.767811 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72d0dee5-5c72-4d20-aa0b-ce70c40d1d85 None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:45:06.768087 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72d0dee5-5c72-4d20-aa0b-ce70c40d1d85 None admin] PATH_INFO: `/v3/domains/b111a0b8935c40a1b0197f851c0c778f` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:45:06.769198 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72d0dee5-5c72-4d20-aa0b-ce70c40d1d85 None admin] RBAC: Authorizing `identity:get_domain(domain_id=b111a0b8935c40a1b0197f851c0c778f)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:45:06.770652 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72d0dee5-5c72-4d20-aa0b-ce70c40d1d85 None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:45:06.772204 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 584/1176] 10.222.0.27 () {56 vars in 1183 bytes} [Tue Aug 30 01:45:06 2022] GET /identity/v3/domains/b111a0b8935c40a1b0197f851c0c778f => generated 267 bytes in 141 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:45:06.777916 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93df6c87-2b48-4cd1-a5f4-3872546455e9 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:45:06.780131 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:45:06.877130 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93df6c87-2b48-4cd1-a5f4-3872546455e9 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:45:06.910096 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99b3d540-6233-4163-bdcc-154f6d817465 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:45:06.910642 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99b3d540-6233-4163-bdcc-154f6d817465 None admin] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:45:06.910919 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99b3d540-6233-4163-bdcc-154f6d817465 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:45:06.911186 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99b3d540-6233-4163-bdcc-154f6d817465 None admin] PATH_INFO: `/v3/users` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:45:06.911837 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99b3d540-6233-4163-bdcc-154f6d817465 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:45:06.913262 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99b3d540-6233-4163-bdcc-154f6d817465 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:45:06.997418 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 593/1177] 10.222.0.27 () {60 vars in 1132 bytes} [Tue Aug 30 01:45:06 2022] POST /identity/v3/users => generated 311 bytes in 220 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 01:45:08.947576 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-076e681a-0bb4-4d05-83aa-59327acfaa74 None None] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:45:08.948130 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-076e681a-0bb4-4d05-83aa-59327acfaa74 None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:45:08.948130 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-076e681a-0bb4-4d05-83aa-59327acfaa74 None None] PATH_INFO: `/` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:45:08.949450 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 585/1178] 10.222.0.27 () {54 vars in 927 bytes} [Tue Aug 30 01:45:08 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 01:45:08.954741 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8eb3c3e7-ef74-4c42-9f13-47f3ff5a87fe None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:45:08.955173 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8eb3c3e7-ef74-4c42-9f13-47f3ff5a87fe None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:45:08.955279 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8eb3c3e7-ef74-4c42-9f13-47f3ff5a87fe None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:45:08.973545 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-8eb3c3e7-ef74-4c42-9f13-47f3ff5a87fe None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 01:45:08.974660 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-8eb3c3e7-ef74-4c42-9f13-47f3ff5a87fe None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:45:09.026164 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:45:09.039773 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 594/1179] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Aug 30 01:45:08 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 30 01:45:09.312012 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-076e681a-0bb4-4d05-83aa-59327acfaa74 None None] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:45:09.410024 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-076e681a-0bb4-4d05-83aa-59327acfaa74 None None] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:45:09.442663 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4658972-df16-477c-898d-db259a8fa05e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:45:09.443271 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4658972-df16-477c-898d-db259a8fa05e None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:45:09.443546 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4658972-df16-477c-898d-db259a8fa05e None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:45:09.443811 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4658972-df16-477c-898d-db259a8fa05e None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:45:09.453437 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4658972-df16-477c-898d-db259a8fa05e None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 30 01:45:09.458553 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application [None req-d4658972-df16-477c-898d-db259a8fa05e None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 30 01:45:09.458553 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 01:45:09.458553 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 01:45:09.458553 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 01:45:09.458553 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 01:45:09.458553 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 01:45:09.458553 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 01:45:09.458553 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 01:45:09.458553 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 01:45:09.458553 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 01:45:09.458553 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 01:45:09.458553 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 01:45:09.458553 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 30 01:45:09.458553 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 30 01:45:09.458553 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 30 01:45:09.458553 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 30 01:45:09.459768 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 30 01:45:09.459768 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 30 01:45:09.459768 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 30 01:45:09.459768 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 30 01:45:09.459768 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 01:45:09.459768 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 01:45:09.459768 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 01:45:09.459768 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 01:45:09.459768 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 01:45:09.459768 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 01:45:09.459768 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 01:45:09.459768 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application with Lock( Aug 30 01:45:09.459768 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 01:45:09.459768 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._enter() Aug 30 01:45:09.459768 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 01:45:09.459768 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 01:45:09.460911 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 01:45:09.460911 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.creator() Aug 30 01:45:09.460911 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 01:45:09.460911 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application created_value = creator( Aug 30 01:45:09.460911 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 30 01:45:09.460911 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 30 01:45:09.460911 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 30 01:45:09.460911 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 30 01:45:09.460911 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 30 01:45:09.460911 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 30 01:45:09.460911 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 30 01:45:09.460911 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application Aug 30 01:45:09.460911 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 586/1180] 10.222.0.27 () {56 vars in 1096 bytes} [Tue Aug 30 01:45:09 2022] GET /identity/v3/roles/admin => generated 83 bytes in 148 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 01:45:09.465739 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8eb3c3e7-ef74-4c42-9f13-47f3ff5a87fe None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:45:09.565283 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8eb3c3e7-ef74-4c42-9f13-47f3ff5a87fe None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:45:09.599990 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55f2e4a6-4446-4c15-a24d-e0b3c63a6c36 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:45:09.600593 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55f2e4a6-4446-4c15-a24d-e0b3c63a6c36 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:45:09.600872 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55f2e4a6-4446-4c15-a24d-e0b3c63a6c36 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:45:09.601397 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55f2e4a6-4446-4c15-a24d-e0b3c63a6c36 None admin] PATH_INFO: `/v3/roles` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:45:09.602086 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55f2e4a6-4446-4c15-a24d-e0b3c63a6c36 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:45:09.602389 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55f2e4a6-4446-4c15-a24d-e0b3c63a6c36 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:45:09.603765 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55f2e4a6-4446-4c15-a24d-e0b3c63a6c36 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:45:09.612309 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 595/1181] 10.222.0.27 () {56 vars in 1099 bytes} [Tue Aug 30 01:45:09 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:45:09.618142 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4658972-df16-477c-898d-db259a8fa05e None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:45:09.648784 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:45:09.722157 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4658972-df16-477c-898d-db259a8fa05e None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:45:09.759073 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46a83fdb-bb8f-434f-8763-ae9c97756fec None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:45:09.759698 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46a83fdb-bb8f-434f-8763-ae9c97756fec None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:45:09.759973 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46a83fdb-bb8f-434f-8763-ae9c97756fec None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:45:09.760365 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46a83fdb-bb8f-434f-8763-ae9c97756fec None admin] PATH_INFO: `/v3/users/67c645a43ee14af9a655e7d595d6dd2a` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:45:09.773506 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46a83fdb-bb8f-434f-8763-ae9c97756fec None admin] RBAC: Authorizing `identity:get_user(user_id=67c645a43ee14af9a655e7d595d6dd2a)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:45:09.774917 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46a83fdb-bb8f-434f-8763-ae9c97756fec None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:45:09.776488 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 587/1182] 10.222.0.27 () {56 vars in 1177 bytes} [Tue Aug 30 01:45:09 2022] GET /identity/v3/users/67c645a43ee14af9a655e7d595d6dd2a => generated 311 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:45:09.783209 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55f2e4a6-4446-4c15-a24d-e0b3c63a6c36 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:45:09.832804 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:45:09.881535 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55f2e4a6-4446-4c15-a24d-e0b3c63a6c36 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:45:09.913463 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de655ee6-f9c7-4d4b-987e-4ba84e8eb59b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:45:09.914064 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de655ee6-f9c7-4d4b-987e-4ba84e8eb59b None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:45:09.914337 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de655ee6-f9c7-4d4b-987e-4ba84e8eb59b None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:45:09.914603 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de655ee6-f9c7-4d4b-987e-4ba84e8eb59b None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:45:09.979187 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de655ee6-f9c7-4d4b-987e-4ba84e8eb59b None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:45:09.980614 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de655ee6-f9c7-4d4b-987e-4ba84e8eb59b None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:45:10.027231 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:45:10.078082 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 596/1183] 10.222.0.27 () {58 vars in 1282 bytes} [Tue Aug 30 01:45:09 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 296 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Aug 30 01:45:10.084278 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46a83fdb-bb8f-434f-8763-ae9c97756fec None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:45:10.170946 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:45:10.180536 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46a83fdb-bb8f-434f-8763-ae9c97756fec None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:45:10.214130 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7918e533-6101-46bd-8583-5b5832ccf1d1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:45:10.214742 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7918e533-6101-46bd-8583-5b5832ccf1d1 None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:45:10.215019 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7918e533-6101-46bd-8583-5b5832ccf1d1 None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:45:10.215284 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7918e533-6101-46bd-8583-5b5832ccf1d1 None admin] PATH_INFO: `/v3/projects/b19e02fc1fda4e9d9a7098b13d0feb04` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:45:10.216368 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7918e533-6101-46bd-8583-5b5832ccf1d1 None admin] RBAC: Authorizing `identity:get_project(project_id=b19e02fc1fda4e9d9a7098b13d0feb04)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:45:10.217798 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7918e533-6101-46bd-8583-5b5832ccf1d1 None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:45:10.219391 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 588/1184] 10.222.0.27 () {56 vars in 1186 bytes} [Tue Aug 30 01:45:10 2022] GET /identity/v3/projects/b19e02fc1fda4e9d9a7098b13d0feb04 => generated 361 bytes in 136 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:45:10.225593 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de655ee6-f9c7-4d4b-987e-4ba84e8eb59b None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:45:10.322379 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de655ee6-f9c7-4d4b-987e-4ba84e8eb59b None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:45:10.355531 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a347db7d-9176-4b54-a057-2dafa9748103 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:45:10.356070 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a347db7d-9176-4b54-a057-2dafa9748103 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:45:10.356348 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a347db7d-9176-4b54-a057-2dafa9748103 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:45:10.356617 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a347db7d-9176-4b54-a057-2dafa9748103 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:45:10.357334 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a347db7d-9176-4b54-a057-2dafa9748103 None admin] RBAC: Adding query filter params (role.id=ead5fac86a0747ea8823a82efcea8ed7, scope.project.id=b19e02fc1fda4e9d9a7098b13d0feb04, user.id=67c645a43ee14af9a655e7d595d6dd2a) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:45:10.357670 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a347db7d-9176-4b54-a057-2dafa9748103 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:45:10.359085 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a347db7d-9176-4b54-a057-2dafa9748103 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:45:10.364933 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 597/1185] 10.222.0.27 () {56 vars in 1374 bytes} [Tue Aug 30 01:45:10 2022] GET /identity/v3/role_assignments?user.id=67c645a43ee14af9a655e7d595d6dd2a&scope.project.id=b19e02fc1fda4e9d9a7098b13d0feb04&role.id=ead5fac86a0747ea8823a82efcea8ed7 => generated 259 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:45:12.233432 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-581ef9a4-a84d-4cfd-af99-6b50e75998eb None None] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:45:12.234011 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-581ef9a4-a84d-4cfd-af99-6b50e75998eb None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:45:12.234011 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-581ef9a4-a84d-4cfd-af99-6b50e75998eb None None] PATH_INFO: `/` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:45:12.235936 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 589/1186] 10.222.0.27 () {54 vars in 927 bytes} [Tue Aug 30 01:45:12 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 01:45:12.241161 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6f40429-3c50-484b-ae30-3d24d7a26b52 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:45:12.241583 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6f40429-3c50-484b-ae30-3d24d7a26b52 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:45:12.241735 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6f40429-3c50-484b-ae30-3d24d7a26b52 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:45:12.259930 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-d6f40429-3c50-484b-ae30-3d24d7a26b52 None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 01:45:12.261106 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-d6f40429-3c50-484b-ae30-3d24d7a26b52 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:45:12.270070 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:45:12.325967 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 598/1187] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Aug 30 01:45:12 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 30 01:45:12.596667 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-581ef9a4-a84d-4cfd-af99-6b50e75998eb None None] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:45:12.705583 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-581ef9a4-a84d-4cfd-af99-6b50e75998eb None None] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:45:12.739392 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d373dd28-e92d-40e1-a92c-7719f60b059c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:45:12.740013 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d373dd28-e92d-40e1-a92c-7719f60b059c None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:45:12.740293 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d373dd28-e92d-40e1-a92c-7719f60b059c None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:45:12.740564 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d373dd28-e92d-40e1-a92c-7719f60b059c None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:45:12.750365 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d373dd28-e92d-40e1-a92c-7719f60b059c None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 30 01:45:12.756097 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application [None req-d373dd28-e92d-40e1-a92c-7719f60b059c None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 30 01:45:12.756097 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 01:45:12.756097 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 01:45:12.756097 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 01:45:12.756097 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 01:45:12.756097 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 01:45:12.756097 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 01:45:12.756097 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 01:45:12.756097 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 01:45:12.756097 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 01:45:12.756097 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 01:45:12.756097 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 01:45:12.756097 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 30 01:45:12.756097 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 30 01:45:12.756097 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 30 01:45:12.756097 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 30 01:45:12.757238 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 30 01:45:12.757238 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 30 01:45:12.757238 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 30 01:45:12.757238 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 30 01:45:12.757238 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 01:45:12.757238 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 01:45:12.757238 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 01:45:12.757238 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 01:45:12.757238 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 01:45:12.757238 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 01:45:12.757238 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 01:45:12.757238 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application with Lock( Aug 30 01:45:12.757238 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 01:45:12.757238 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._enter() Aug 30 01:45:12.757238 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 01:45:12.757238 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 01:45:12.758390 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 01:45:12.758390 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.creator() Aug 30 01:45:12.758390 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 01:45:12.758390 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application created_value = creator( Aug 30 01:45:12.758390 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 30 01:45:12.758390 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 30 01:45:12.758390 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 30 01:45:12.758390 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 30 01:45:12.758390 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 30 01:45:12.758390 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 30 01:45:12.758390 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 30 01:45:12.758390 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application Aug 30 01:45:12.758390 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 590/1188] 10.222.0.27 () {56 vars in 1096 bytes} [Tue Aug 30 01:45:12 2022] GET /identity/v3/roles/admin => generated 83 bytes in 161 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 01:45:12.763161 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6f40429-3c50-484b-ae30-3d24d7a26b52 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:45:12.859973 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6f40429-3c50-484b-ae30-3d24d7a26b52 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:45:12.893501 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08736d52-2dc4-4f03-8e7f-d3409dc5b79a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:45:12.894097 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08736d52-2dc4-4f03-8e7f-d3409dc5b79a None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:45:12.894377 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08736d52-2dc4-4f03-8e7f-d3409dc5b79a None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:45:12.894648 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08736d52-2dc4-4f03-8e7f-d3409dc5b79a None admin] PATH_INFO: `/v3/roles` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:45:12.895302 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08736d52-2dc4-4f03-8e7f-d3409dc5b79a None admin] RBAC: Adding query filter params (name=admin) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:45:12.895604 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08736d52-2dc4-4f03-8e7f-d3409dc5b79a None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:45:12.896966 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08736d52-2dc4-4f03-8e7f-d3409dc5b79a None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:45:12.904721 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 599/1189] 10.222.0.27 () {56 vars in 1099 bytes} [Tue Aug 30 01:45:12 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 142 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:45:12.910713 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d373dd28-e92d-40e1-a92c-7719f60b059c None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:45:12.930185 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:45:13.009686 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d373dd28-e92d-40e1-a92c-7719f60b059c None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:45:13.045935 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59971b50-8de2-49f5-998e-791f61ddc1df None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:45:13.046556 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59971b50-8de2-49f5-998e-791f61ddc1df None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:45:13.046837 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59971b50-8de2-49f5-998e-791f61ddc1df None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:45:13.047105 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59971b50-8de2-49f5-998e-791f61ddc1df None admin] PATH_INFO: `/v3/users/67c645a43ee14af9a655e7d595d6dd2a` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:45:13.048846 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59971b50-8de2-49f5-998e-791f61ddc1df None admin] RBAC: Authorizing `identity:get_user(user_id=67c645a43ee14af9a655e7d595d6dd2a)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:45:13.050300 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59971b50-8de2-49f5-998e-791f61ddc1df None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:45:13.051872 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 591/1190] 10.222.0.27 () {56 vars in 1177 bytes} [Tue Aug 30 01:45:12 2022] GET /identity/v3/users/67c645a43ee14af9a655e7d595d6dd2a => generated 311 bytes in 142 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:45:13.057903 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08736d52-2dc4-4f03-8e7f-d3409dc5b79a None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:45:13.067587 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:45:13.155029 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08736d52-2dc4-4f03-8e7f-d3409dc5b79a None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:45:13.187491 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a9af7f4-b9fc-4493-81ab-f9d4bcb41b15 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:45:13.188105 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a9af7f4-b9fc-4493-81ab-f9d4bcb41b15 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:45:13.189043 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a9af7f4-b9fc-4493-81ab-f9d4bcb41b15 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:45:13.189318 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a9af7f4-b9fc-4493-81ab-f9d4bcb41b15 None admin] PATH_INFO: `/v3/projects/b19e02fc1fda4e9d9a7098b13d0feb04` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:45:13.190502 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a9af7f4-b9fc-4493-81ab-f9d4bcb41b15 None admin] RBAC: Authorizing `identity:get_project(project_id=b19e02fc1fda4e9d9a7098b13d0feb04)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:45:13.191959 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a9af7f4-b9fc-4493-81ab-f9d4bcb41b15 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:45:13.193598 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 600/1191] 10.222.0.27 () {56 vars in 1186 bytes} [Tue Aug 30 01:45:13 2022] GET /identity/v3/projects/b19e02fc1fda4e9d9a7098b13d0feb04 => generated 361 bytes in 136 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:45:13.199460 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59971b50-8de2-49f5-998e-791f61ddc1df None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:45:13.281793 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:45:13.297672 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59971b50-8de2-49f5-998e-791f61ddc1df None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:45:13.332187 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-defe7e5e-a8b7-4069-8c80-ac58edf250fa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:45:13.332825 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-defe7e5e-a8b7-4069-8c80-ac58edf250fa None admin] REQUEST_METHOD: `PUT` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:45:13.333104 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-defe7e5e-a8b7-4069-8c80-ac58edf250fa None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:45:13.333374 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-defe7e5e-a8b7-4069-8c80-ac58edf250fa None admin] PATH_INFO: `/v3/projects/b19e02fc1fda4e9d9a7098b13d0feb04/users/67c645a43ee14af9a655e7d595d6dd2a/roles/ead5fac86a0747ea8823a82efcea8ed7` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:45:13.335254 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-defe7e5e-a8b7-4069-8c80-ac58edf250fa None admin] RBAC: Authorizing `identity:create_grant(project_id=b19e02fc1fda4e9d9a7098b13d0feb04, user_id=67c645a43ee14af9a655e7d595d6dd2a, role_id=ead5fac86a0747ea8823a82efcea8ed7)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:45:13.336661 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-defe7e5e-a8b7-4069-8c80-ac58edf250fa None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:45:13.410823 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 592/1192] 10.222.0.27 () {58 vars in 1439 bytes} [Tue Aug 30 01:45:13 2022] PUT /identity/v3/projects/b19e02fc1fda4e9d9a7098b13d0feb04/users/67c645a43ee14af9a655e7d595d6dd2a/roles/ead5fac86a0747ea8823a82efcea8ed7 => generated 0 bytes in 212 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 01:45:15.291016 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-143ef8ef-62cc-44a2-bd3f-6bb6fb2a4867 None None] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:45:15.291544 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-143ef8ef-62cc-44a2-bd3f-6bb6fb2a4867 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:45:15.291544 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-143ef8ef-62cc-44a2-bd3f-6bb6fb2a4867 None None] PATH_INFO: `/` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:45:15.292897 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 601/1193] 10.222.0.27 () {54 vars in 927 bytes} [Tue Aug 30 01:45:15 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 01:45:15.298708 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cde5d29a-230a-4317-8bfb-92cf587c88b2 None None] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:45:15.299147 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cde5d29a-230a-4317-8bfb-92cf587c88b2 None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:45:15.299344 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cde5d29a-230a-4317-8bfb-92cf587c88b2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:45:15.317884 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.auth.core [None req-cde5d29a-230a-4317-8bfb-92cf587c88b2 None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81578) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 01:45:15.319065 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-cde5d29a-230a-4317-8bfb-92cf587c88b2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:45:15.384695 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 593/1194] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Aug 30 01:45:15 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 30 01:45:15.652142 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-143ef8ef-62cc-44a2-bd3f-6bb6fb2a4867 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:45:15.728837 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:45:15.753868 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-143ef8ef-62cc-44a2-bd3f-6bb6fb2a4867 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:45:15.789863 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15ef12a9-3d7e-4574-a094-c49df3fa9dfd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:45:15.790480 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15ef12a9-3d7e-4574-a094-c49df3fa9dfd None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:45:15.790754 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15ef12a9-3d7e-4574-a094-c49df3fa9dfd None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:45:15.791021 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15ef12a9-3d7e-4574-a094-c49df3fa9dfd None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:45:15.800684 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15ef12a9-3d7e-4574-a094-c49df3fa9dfd None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 30 01:45:15.805366 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application [None req-15ef12a9-3d7e-4574-a094-c49df3fa9dfd None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 30 01:45:15.805366 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 01:45:15.805366 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 01:45:15.805366 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 01:45:15.805366 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 01:45:15.805366 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 01:45:15.805366 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 01:45:15.805366 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 01:45:15.805366 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 01:45:15.805366 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 01:45:15.805366 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 01:45:15.805366 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 01:45:15.805366 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 30 01:45:15.805366 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 30 01:45:15.805366 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 30 01:45:15.805366 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 30 01:45:15.806507 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 30 01:45:15.806507 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 30 01:45:15.806507 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 30 01:45:15.806507 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 30 01:45:15.806507 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 01:45:15.806507 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 01:45:15.806507 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 01:45:15.806507 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 01:45:15.806507 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 01:45:15.806507 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 01:45:15.806507 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 01:45:15.806507 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application with Lock( Aug 30 01:45:15.806507 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 01:45:15.806507 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._enter() Aug 30 01:45:15.806507 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 01:45:15.806507 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 01:45:15.807564 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 01:45:15.807564 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.creator() Aug 30 01:45:15.807564 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 01:45:15.807564 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application created_value = creator( Aug 30 01:45:15.807564 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 30 01:45:15.807564 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 30 01:45:15.807564 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 30 01:45:15.807564 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 30 01:45:15.807564 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 30 01:45:15.807564 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 30 01:45:15.807564 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 30 01:45:15.807564 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Aug 30 01:45:15.807564 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 602/1195] 10.222.0.27 () {56 vars in 1096 bytes} [Tue Aug 30 01:45:15 2022] GET /identity/v3/roles/admin => generated 83 bytes in 155 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 01:45:15.812605 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cde5d29a-230a-4317-8bfb-92cf587c88b2 None None] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:45:15.886768 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:45:15.908927 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cde5d29a-230a-4317-8bfb-92cf587c88b2 None None] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:45:15.942259 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc9ba54b-52eb-468d-a5a3-c126401c7432 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:45:15.942848 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc9ba54b-52eb-468d-a5a3-c126401c7432 None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:45:15.943125 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc9ba54b-52eb-468d-a5a3-c126401c7432 None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:45:15.943396 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc9ba54b-52eb-468d-a5a3-c126401c7432 None admin] PATH_INFO: `/v3/roles` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:45:15.944046 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc9ba54b-52eb-468d-a5a3-c126401c7432 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81578) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:45:15.944345 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc9ba54b-52eb-468d-a5a3-c126401c7432 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:45:15.945720 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc9ba54b-52eb-468d-a5a3-c126401c7432 None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:45:15.953583 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 594/1196] 10.222.0.27 () {56 vars in 1099 bytes} [Tue Aug 30 01:45:15 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 141 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:45:15.959337 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15ef12a9-3d7e-4574-a094-c49df3fa9dfd None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:45:16.056518 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15ef12a9-3d7e-4574-a094-c49df3fa9dfd None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:45:16.089838 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-382f7b0c-b982-4c2e-af9d-922eb70d6a4b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:45:16.090650 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-382f7b0c-b982-4c2e-af9d-922eb70d6a4b None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:45:16.091657 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-382f7b0c-b982-4c2e-af9d-922eb70d6a4b None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:45:16.091931 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-382f7b0c-b982-4c2e-af9d-922eb70d6a4b None admin] PATH_INFO: `/v3/users/67c645a43ee14af9a655e7d595d6dd2a` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:45:16.093047 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-382f7b0c-b982-4c2e-af9d-922eb70d6a4b None admin] RBAC: Authorizing `identity:get_user(user_id=67c645a43ee14af9a655e7d595d6dd2a)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:45:16.094630 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-382f7b0c-b982-4c2e-af9d-922eb70d6a4b None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:45:16.096207 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 603/1197] 10.222.0.27 () {56 vars in 1177 bytes} [Tue Aug 30 01:45:15 2022] GET /identity/v3/users/67c645a43ee14af9a655e7d595d6dd2a => generated 311 bytes in 137 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 30 01:45:16.102303 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc9ba54b-52eb-468d-a5a3-c126401c7432 None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:45:16.198687 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc9ba54b-52eb-468d-a5a3-c126401c7432 None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:45:16.231473 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0e8186d-d2eb-4139-8a7b-13f919804f70 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:45:16.232061 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0e8186d-d2eb-4139-8a7b-13f919804f70 None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:45:16.232339 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0e8186d-d2eb-4139-8a7b-13f919804f70 None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:45:16.232610 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0e8186d-d2eb-4139-8a7b-13f919804f70 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:45:16.240793 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:45:16.297199 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0e8186d-d2eb-4139-8a7b-13f919804f70 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:45:16.298743 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0e8186d-d2eb-4139-8a7b-13f919804f70 None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:45:16.394037 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 595/1198] 10.222.0.27 () {58 vars in 1282 bytes} [Tue Aug 30 01:45:16 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 292 msecs (HTTP/1.1 200) 6 headers in 359 bytes (2 switches on core 0) Aug 30 01:45:16.399996 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-382f7b0c-b982-4c2e-af9d-922eb70d6a4b None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:45:16.402916 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:45:16.497511 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-382f7b0c-b982-4c2e-af9d-922eb70d6a4b None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:45:16.530438 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a90e07d7-7b0c-48c4-9f55-7f24bdfb4759 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:45:16.531050 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a90e07d7-7b0c-48c4-9f55-7f24bdfb4759 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:45:16.531324 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a90e07d7-7b0c-48c4-9f55-7f24bdfb4759 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:45:16.531589 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a90e07d7-7b0c-48c4-9f55-7f24bdfb4759 None admin] PATH_INFO: `/v3/projects/b19e02fc1fda4e9d9a7098b13d0feb04` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:45:16.532887 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a90e07d7-7b0c-48c4-9f55-7f24bdfb4759 None admin] RBAC: Authorizing `identity:get_project(project_id=b19e02fc1fda4e9d9a7098b13d0feb04)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:45:16.534344 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a90e07d7-7b0c-48c4-9f55-7f24bdfb4759 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:45:16.535953 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 604/1199] 10.222.0.27 () {56 vars in 1186 bytes} [Tue Aug 30 01:45:16 2022] GET /identity/v3/projects/b19e02fc1fda4e9d9a7098b13d0feb04 => generated 361 bytes in 136 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:45:16.541945 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0e8186d-d2eb-4139-8a7b-13f919804f70 None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:45:16.580193 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:45:16.642213 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0e8186d-d2eb-4139-8a7b-13f919804f70 None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:45:16.685834 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7e1be1e-f1a1-45bd-bb79-d16dad105b25 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:45:16.686427 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7e1be1e-f1a1-45bd-bb79-d16dad105b25 None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:45:16.686701 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7e1be1e-f1a1-45bd-bb79-d16dad105b25 None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:45:16.687075 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7e1be1e-f1a1-45bd-bb79-d16dad105b25 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:45:16.688759 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7e1be1e-f1a1-45bd-bb79-d16dad105b25 None admin] RBAC: Adding query filter params (role.id=ead5fac86a0747ea8823a82efcea8ed7, scope.project.id=b19e02fc1fda4e9d9a7098b13d0feb04, user.id=67c645a43ee14af9a655e7d595d6dd2a) {{(pid=81578) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:45:16.689233 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7e1be1e-f1a1-45bd-bb79-d16dad105b25 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:45:16.691507 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7e1be1e-f1a1-45bd-bb79-d16dad105b25 None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:45:16.699911 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 596/1200] 10.222.0.27 () {56 vars in 1374 bytes} [Tue Aug 30 01:45:16 2022] GET /identity/v3/role_assignments?user.id=67c645a43ee14af9a655e7d595d6dd2a&scope.project.id=b19e02fc1fda4e9d9a7098b13d0feb04&role.id=ead5fac86a0747ea8823a82efcea8ed7 => generated 608 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:45:26.526426 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=65 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:45:26.695882 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=49 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:46:29.773642 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6b63391-d1d4-4372-afbf-1201f30af2cd None None] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:46:29.774205 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6b63391-d1d4-4372-afbf-1201f30af2cd None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:46:29.774205 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6b63391-d1d4-4372-afbf-1201f30af2cd None None] PATH_INFO: `/` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:46:29.775526 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 605/1201] 10.222.0.27 () {54 vars in 927 bytes} [Tue Aug 30 01:46:29 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 01:46:29.780732 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1e27f1f-f1c1-4f20-b492-72e58e6e63e4 None None] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:46:29.781130 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1e27f1f-f1c1-4f20-b492-72e58e6e63e4 None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:46:29.781265 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1e27f1f-f1c1-4f20-b492-72e58e6e63e4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:46:29.799880 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.auth.core [None req-b1e27f1f-f1c1-4f20-b492-72e58e6e63e4 None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81578) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 01:46:29.801143 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-b1e27f1f-f1c1-4f20-b492-72e58e6e63e4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:46:29.871584 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 597/1202] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Aug 30 01:46:29 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 30 01:46:30.147059 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6b63391-d1d4-4372-afbf-1201f30af2cd None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:46:30.248685 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6b63391-d1d4-4372-afbf-1201f30af2cd None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:46:30.288026 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a90fc77-074b-4df8-9c87-45d5bb4c24e4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:46:30.288656 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a90fc77-074b-4df8-9c87-45d5bb4c24e4 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:46:30.288930 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a90fc77-074b-4df8-9c87-45d5bb4c24e4 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:46:30.289200 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a90fc77-074b-4df8-9c87-45d5bb4c24e4 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:46:30.299379 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a90fc77-074b-4df8-9c87-45d5bb4c24e4 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:46:30.300850 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a90fc77-074b-4df8-9c87-45d5bb4c24e4 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:46:30.316004 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application [None req-5a90fc77-074b-4df8-9c87-45d5bb4c24e4 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 01:46:30.316004 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 01:46:30.316004 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 01:46:30.316004 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 01:46:30.316004 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 01:46:30.316004 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 01:46:30.316004 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 01:46:30.316004 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 01:46:30.316004 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 01:46:30.316004 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 01:46:30.316004 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 01:46:30.316004 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 01:46:30.316004 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 30 01:46:30.316004 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 30 01:46:30.316004 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 30 01:46:30.316004 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 30 01:46:30.317937 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 01:46:30.317937 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 01:46:30.317937 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 01:46:30.317937 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 01:46:30.317937 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 01:46:30.317937 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 01:46:30.317937 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 01:46:30.317937 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application with Lock( Aug 30 01:46:30.317937 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 01:46:30.317937 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._enter() Aug 30 01:46:30.317937 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 01:46:30.317937 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 01:46:30.317937 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 01:46:30.317937 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.creator() Aug 30 01:46:30.317937 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 01:46:30.317937 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application created_value = creator( Aug 30 01:46:30.317937 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 30 01:46:30.319264 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 30 01:46:30.319264 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 01:46:30.319264 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Aug 30 01:46:30.319264 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 606/1203] 10.222.0.27 () {56 vars in 1108 bytes} [Tue Aug 30 01:46:30 2022] GET /identity/v3/domains/Default => generated 87 bytes in 172 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 30 01:46:30.326334 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1e27f1f-f1c1-4f20-b492-72e58e6e63e4 None None] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:46:30.426517 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1e27f1f-f1c1-4f20-b492-72e58e6e63e4 None None] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:46:30.454821 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:46:30.460645 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1a1da94-3d8f-405e-93c9-c808f1b8d450 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:46:30.461364 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1a1da94-3d8f-405e-93c9-c808f1b8d450 None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:46:30.461682 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1a1da94-3d8f-405e-93c9-c808f1b8d450 None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:46:30.461952 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1a1da94-3d8f-405e-93c9-c808f1b8d450 None admin] PATH_INFO: `/v3/domains` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:46:30.462588 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1a1da94-3d8f-405e-93c9-c808f1b8d450 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81578) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:46:30.462889 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1a1da94-3d8f-405e-93c9-c808f1b8d450 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:46:30.464251 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1a1da94-3d8f-405e-93c9-c808f1b8d450 None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:46:30.476204 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 598/1204] 10.222.0.27 () {56 vars in 1109 bytes} [Tue Aug 30 01:46:30 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:46:30.482110 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a90fc77-074b-4df8-9c87-45d5bb4c24e4 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:46:30.534561 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:46:30.581984 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a90fc77-074b-4df8-9c87-45d5bb4c24e4 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:46:30.616907 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a04c3504-5c6b-46f6-be5c-7eae2973e2d6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:46:30.618080 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a04c3504-5c6b-46f6-be5c-7eae2973e2d6 None admin] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:46:30.618080 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a04c3504-5c6b-46f6-be5c-7eae2973e2d6 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:46:30.618311 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a04c3504-5c6b-46f6-be5c-7eae2973e2d6 None admin] PATH_INFO: `/v3/users` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:46:30.618777 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a04c3504-5c6b-46f6-be5c-7eae2973e2d6 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:46:30.620252 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a04c3504-5c6b-46f6-be5c-7eae2973e2d6 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:46:30.692115 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 607/1205] 10.222.0.27 () {60 vars in 1132 bytes} [Tue Aug 30 01:46:30 2022] POST /identity/v3/users => generated 251 bytes in 210 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 01:46:32.656207 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15e01c37-27c1-4138-8a0e-4c3abd577728 None None] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:46:32.656798 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15e01c37-27c1-4138-8a0e-4c3abd577728 None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:46:32.656798 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15e01c37-27c1-4138-8a0e-4c3abd577728 None None] PATH_INFO: `/` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:46:32.658227 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 599/1206] 10.222.0.27 () {54 vars in 927 bytes} [Tue Aug 30 01:46:32 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 01:46:32.664198 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7bd05d5-9c85-4293-b7bf-db36cfcb1f5a None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:46:32.664620 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7bd05d5-9c85-4293-b7bf-db36cfcb1f5a None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:46:32.670717 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7bd05d5-9c85-4293-b7bf-db36cfcb1f5a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:46:32.684640 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-c7bd05d5-9c85-4293-b7bf-db36cfcb1f5a None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 01:46:32.685920 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-c7bd05d5-9c85-4293-b7bf-db36cfcb1f5a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:46:32.754355 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 608/1207] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Aug 30 01:46:32 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 30 01:46:33.023860 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15e01c37-27c1-4138-8a0e-4c3abd577728 None None] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:46:33.124721 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15e01c37-27c1-4138-8a0e-4c3abd577728 None None] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:46:33.158052 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec944c0a-e28a-4f18-ac67-a06c09cfad6c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:46:33.158669 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec944c0a-e28a-4f18-ac67-a06c09cfad6c None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:46:33.158940 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec944c0a-e28a-4f18-ac67-a06c09cfad6c None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:46:33.159202 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec944c0a-e28a-4f18-ac67-a06c09cfad6c None admin] PATH_INFO: `/v3/roles/service` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:46:33.169440 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec944c0a-e28a-4f18-ac67-a06c09cfad6c None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 30 01:46:33.174195 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application [None req-ec944c0a-e28a-4f18-ac67-a06c09cfad6c None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Aug 30 01:46:33.174195 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 01:46:33.174195 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 01:46:33.174195 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 01:46:33.174195 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 01:46:33.174195 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 01:46:33.174195 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 01:46:33.174195 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 01:46:33.174195 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 01:46:33.174195 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 01:46:33.174195 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 01:46:33.174195 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 01:46:33.174195 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 30 01:46:33.174195 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 30 01:46:33.174195 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 30 01:46:33.174195 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 30 01:46:33.175327 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 30 01:46:33.175327 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 30 01:46:33.175327 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 30 01:46:33.175327 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 30 01:46:33.175327 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 01:46:33.175327 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 01:46:33.175327 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 01:46:33.175327 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 01:46:33.175327 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 01:46:33.175327 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 01:46:33.175327 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 01:46:33.175327 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application with Lock( Aug 30 01:46:33.175327 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 01:46:33.175327 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._enter() Aug 30 01:46:33.175327 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 01:46:33.175327 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 01:46:33.177031 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 01:46:33.177031 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.creator() Aug 30 01:46:33.177031 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 01:46:33.177031 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application created_value = creator( Aug 30 01:46:33.177031 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 30 01:46:33.177031 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 30 01:46:33.177031 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 30 01:46:33.177031 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 30 01:46:33.177031 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 30 01:46:33.177031 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 30 01:46:33.177031 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Aug 30 01:46:33.177031 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application Aug 30 01:46:33.177031 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 600/1208] 10.222.0.27 () {56 vars in 1102 bytes} [Tue Aug 30 01:46:33 2022] GET /identity/v3/roles/service => generated 85 bytes in 152 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 30 01:46:33.181811 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7bd05d5-9c85-4293-b7bf-db36cfcb1f5a None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:46:33.221158 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:46:33.279420 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7bd05d5-9c85-4293-b7bf-db36cfcb1f5a None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:46:33.315190 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2600575b-5943-49f2-b6e9-a0a338b7eaa7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:46:33.315796 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2600575b-5943-49f2-b6e9-a0a338b7eaa7 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:46:33.316072 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2600575b-5943-49f2-b6e9-a0a338b7eaa7 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:46:33.316341 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2600575b-5943-49f2-b6e9-a0a338b7eaa7 None admin] PATH_INFO: `/v3/roles` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:46:33.317006 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2600575b-5943-49f2-b6e9-a0a338b7eaa7 None admin] RBAC: Adding query filter params (name=service) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:46:33.318049 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2600575b-5943-49f2-b6e9-a0a338b7eaa7 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:46:33.319420 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2600575b-5943-49f2-b6e9-a0a338b7eaa7 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:46:33.327074 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 609/1209] 10.222.0.27 () {56 vars in 1103 bytes} [Tue Aug 30 01:46:33 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:46:33.332817 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec944c0a-e28a-4f18-ac67-a06c09cfad6c None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:46:33.383068 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:46:33.431116 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec944c0a-e28a-4f18-ac67-a06c09cfad6c None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:46:33.463371 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-342ee2dd-091a-4840-ae35-7ed7b80b966d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:46:33.463988 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-342ee2dd-091a-4840-ae35-7ed7b80b966d None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:46:33.464265 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-342ee2dd-091a-4840-ae35-7ed7b80b966d None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:46:33.464532 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-342ee2dd-091a-4840-ae35-7ed7b80b966d None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:46:33.475814 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-342ee2dd-091a-4840-ae35-7ed7b80b966d None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:46:33.477237 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-342ee2dd-091a-4840-ae35-7ed7b80b966d None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:46:33.491931 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application [None req-342ee2dd-091a-4840-ae35-7ed7b80b966d None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 01:46:33.491931 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 01:46:33.491931 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 01:46:33.491931 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 01:46:33.491931 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 01:46:33.491931 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 01:46:33.491931 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 01:46:33.491931 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 01:46:33.491931 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 01:46:33.491931 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 01:46:33.491931 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 01:46:33.491931 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 01:46:33.491931 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 30 01:46:33.491931 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 30 01:46:33.491931 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 30 01:46:33.491931 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 30 01:46:33.493058 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 01:46:33.493058 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 01:46:33.493058 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 01:46:33.493058 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 01:46:33.493058 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 01:46:33.493058 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 01:46:33.493058 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 01:46:33.493058 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application with Lock( Aug 30 01:46:33.493058 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 01:46:33.493058 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._enter() Aug 30 01:46:33.493058 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 01:46:33.493058 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 01:46:33.493058 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 01:46:33.493058 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.creator() Aug 30 01:46:33.493058 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 01:46:33.493058 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application created_value = creator( Aug 30 01:46:33.493058 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 30 01:46:33.494202 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 30 01:46:33.494202 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 01:46:33.494202 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application Aug 30 01:46:33.494202 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 601/1210] 10.222.0.27 () {56 vars in 1108 bytes} [Tue Aug 30 01:46:33 2022] GET /identity/v3/domains/Default => generated 87 bytes in 161 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 01:46:33.499063 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2600575b-5943-49f2-b6e9-a0a338b7eaa7 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:46:33.593515 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:46:33.596349 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2600575b-5943-49f2-b6e9-a0a338b7eaa7 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:46:33.631097 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-086f6863-091f-48db-b234-a790fcd0e495 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:46:33.631754 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-086f6863-091f-48db-b234-a790fcd0e495 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:46:33.632074 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-086f6863-091f-48db-b234-a790fcd0e495 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:46:33.632342 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-086f6863-091f-48db-b234-a790fcd0e495 None admin] PATH_INFO: `/v3/domains` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:46:33.632981 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-086f6863-091f-48db-b234-a790fcd0e495 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:46:33.633279 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-086f6863-091f-48db-b234-a790fcd0e495 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:46:33.634714 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-086f6863-091f-48db-b234-a790fcd0e495 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:46:33.646398 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 610/1211] 10.222.0.27 () {56 vars in 1109 bytes} [Tue Aug 30 01:46:33 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:46:33.652133 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-342ee2dd-091a-4840-ae35-7ed7b80b966d None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:46:33.753086 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-342ee2dd-091a-4840-ae35-7ed7b80b966d None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:46:33.755307 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:46:33.786462 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd70800b-609f-4c2a-bdfe-be183a30eaf1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:46:33.787076 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd70800b-609f-4c2a-bdfe-be183a30eaf1 None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:46:33.787351 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd70800b-609f-4c2a-bdfe-be183a30eaf1 None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:46:33.787618 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd70800b-609f-4c2a-bdfe-be183a30eaf1 None admin] PATH_INFO: `/v3/users/placement` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:46:33.795832 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd70800b-609f-4c2a-bdfe-be183a30eaf1 None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:46:33.797246 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd70800b-609f-4c2a-bdfe-be183a30eaf1 None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:46:33.808571 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application [None req-bd70800b-609f-4c2a-bdfe-be183a30eaf1 None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Aug 30 01:46:33.808571 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 01:46:33.808571 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 01:46:33.808571 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 01:46:33.808571 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 01:46:33.808571 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 01:46:33.808571 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 01:46:33.808571 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 01:46:33.808571 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 01:46:33.808571 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 01:46:33.808571 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 01:46:33.808571 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 01:46:33.808571 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 30 01:46:33.808571 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 30 01:46:33.808571 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 30 01:46:33.808571 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 30 01:46:33.814043 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 01:46:33.814043 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 01:46:33.814043 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 30 01:46:33.814043 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 30 01:46:33.814043 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 30 01:46:33.814043 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 30 01:46:33.814043 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 01:46:33.814043 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 01:46:33.814043 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 01:46:33.814043 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 01:46:33.814043 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 01:46:33.814043 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application with Lock( Aug 30 01:46:33.814043 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 01:46:33.814043 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._enter() Aug 30 01:46:33.814043 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 01:46:33.814043 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 01:46:33.814043 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 01:46:33.814043 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.creator() Aug 30 01:46:33.815257 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 01:46:33.815257 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application created_value = creator( Aug 30 01:46:33.815257 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 30 01:46:33.815257 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 30 01:46:33.815257 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 30 01:46:33.815257 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 30 01:46:33.815257 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 30 01:46:33.815257 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 30 01:46:33.815257 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Aug 30 01:46:33.815257 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application Aug 30 01:46:33.815257 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 602/1212] 10.222.0.27 () {56 vars in 1108 bytes} [Tue Aug 30 01:46:33 2022] GET /identity/v3/users/placement => generated 87 bytes in 158 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 01:46:33.816016 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-086f6863-091f-48db-b234-a790fcd0e495 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:46:33.916830 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-086f6863-091f-48db-b234-a790fcd0e495 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:46:33.954055 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c6364fc-7376-40d5-b36f-31d8f9789162 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:46:33.954656 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c6364fc-7376-40d5-b36f-31d8f9789162 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:46:33.954938 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c6364fc-7376-40d5-b36f-31d8f9789162 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:46:33.955212 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c6364fc-7376-40d5-b36f-31d8f9789162 None admin] PATH_INFO: `/v3/users` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:46:33.955943 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c6364fc-7376-40d5-b36f-31d8f9789162 None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:46:33.956246 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c6364fc-7376-40d5-b36f-31d8f9789162 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:46:33.957729 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c6364fc-7376-40d5-b36f-31d8f9789162 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:46:33.980817 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 611/1213] 10.222.0.27 () {56 vars in 1143 bytes} [Tue Aug 30 01:46:33 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:46:33.986629 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd70800b-609f-4c2a-bdfe-be183a30eaf1 None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:46:34.087641 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd70800b-609f-4c2a-bdfe-be183a30eaf1 None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:46:34.119818 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5edff18-e754-4add-b62f-ac4fd2c454b6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:46:34.120413 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5edff18-e754-4add-b62f-ac4fd2c454b6 None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:46:34.120689 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5edff18-e754-4add-b62f-ac4fd2c454b6 None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:46:34.120954 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5edff18-e754-4add-b62f-ac4fd2c454b6 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:46:34.160221 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:46:34.196832 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5edff18-e754-4add-b62f-ac4fd2c454b6 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:46:34.198240 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5edff18-e754-4add-b62f-ac4fd2c454b6 None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:46:34.305664 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 603/1214] 10.222.0.27 () {58 vars in 1282 bytes} [Tue Aug 30 01:46:33 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 319 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Aug 30 01:46:34.312262 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c6364fc-7376-40d5-b36f-31d8f9789162 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:46:34.334153 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:46:34.421725 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c6364fc-7376-40d5-b36f-31d8f9789162 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:46:34.455731 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-657b6c58-89fd-4684-98fb-adbad97edb5f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:46:34.456445 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-657b6c58-89fd-4684-98fb-adbad97edb5f None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:46:34.456725 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-657b6c58-89fd-4684-98fb-adbad97edb5f None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:46:34.456998 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-657b6c58-89fd-4684-98fb-adbad97edb5f None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:46:34.467342 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-657b6c58-89fd-4684-98fb-adbad97edb5f None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:46:34.468941 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-657b6c58-89fd-4684-98fb-adbad97edb5f None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:46:34.485029 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application [None req-657b6c58-89fd-4684-98fb-adbad97edb5f None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 01:46:34.485029 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 01:46:34.485029 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 01:46:34.485029 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 01:46:34.485029 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 01:46:34.485029 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 01:46:34.485029 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 01:46:34.485029 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 01:46:34.485029 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 01:46:34.485029 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 01:46:34.485029 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 01:46:34.485029 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 01:46:34.485029 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 30 01:46:34.485029 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 30 01:46:34.485029 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 30 01:46:34.485029 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 30 01:46:34.486216 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 01:46:34.486216 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 01:46:34.486216 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 01:46:34.486216 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 01:46:34.486216 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 01:46:34.486216 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 01:46:34.486216 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 01:46:34.486216 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application with Lock( Aug 30 01:46:34.486216 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 01:46:34.486216 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._enter() Aug 30 01:46:34.486216 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 01:46:34.486216 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 01:46:34.486216 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 01:46:34.486216 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.creator() Aug 30 01:46:34.486216 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 01:46:34.486216 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application created_value = creator( Aug 30 01:46:34.486216 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 30 01:46:34.487534 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 30 01:46:34.487534 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 01:46:34.487534 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Aug 30 01:46:34.487534 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 612/1215] 10.222.0.27 () {56 vars in 1108 bytes} [Tue Aug 30 01:46:34 2022] GET /identity/v3/domains/Default => generated 87 bytes in 175 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 01:46:34.492876 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5edff18-e754-4add-b62f-ac4fd2c454b6 None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:46:34.554004 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:46:34.591963 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5edff18-e754-4add-b62f-ac4fd2c454b6 None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:46:34.626906 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7304ea4e-37b3-43db-a841-59128e1a5d48 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:46:34.627497 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7304ea4e-37b3-43db-a841-59128e1a5d48 None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:46:34.627773 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7304ea4e-37b3-43db-a841-59128e1a5d48 None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:46:34.628038 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7304ea4e-37b3-43db-a841-59128e1a5d48 None admin] PATH_INFO: `/v3/domains` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:46:34.628668 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7304ea4e-37b3-43db-a841-59128e1a5d48 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81578) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:46:34.629026 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7304ea4e-37b3-43db-a841-59128e1a5d48 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:46:34.630660 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7304ea4e-37b3-43db-a841-59128e1a5d48 None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:46:34.643233 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 604/1216] 10.222.0.27 () {56 vars in 1109 bytes} [Tue Aug 30 01:46:34 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:46:34.649678 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-657b6c58-89fd-4684-98fb-adbad97edb5f None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:46:34.721497 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:46:34.752431 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-657b6c58-89fd-4684-98fb-adbad97edb5f None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:46:34.785060 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c02a8e7c-be15-49ca-9a62-386b17cfc657 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:46:34.785717 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c02a8e7c-be15-49ca-9a62-386b17cfc657 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:46:34.786010 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c02a8e7c-be15-49ca-9a62-386b17cfc657 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:46:34.786282 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c02a8e7c-be15-49ca-9a62-386b17cfc657 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:46:34.792818 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c02a8e7c-be15-49ca-9a62-386b17cfc657 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:46:34.794879 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c02a8e7c-be15-49ca-9a62-386b17cfc657 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:46:34.806605 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application [None req-c02a8e7c-be15-49ca-9a62-386b17cfc657 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 30 01:46:34.806605 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 01:46:34.806605 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 01:46:34.806605 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 01:46:34.806605 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 01:46:34.806605 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 01:46:34.806605 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 01:46:34.806605 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 01:46:34.806605 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 01:46:34.806605 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 01:46:34.806605 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 01:46:34.806605 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 01:46:34.806605 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 30 01:46:34.806605 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 30 01:46:34.806605 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 30 01:46:34.807714 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 30 01:46:34.807714 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 01:46:34.807714 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 01:46:34.807714 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 01:46:34.807714 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 01:46:34.807714 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 01:46:34.807714 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 01:46:34.807714 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 01:46:34.807714 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application with Lock( Aug 30 01:46:34.807714 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 01:46:34.807714 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._enter() Aug 30 01:46:34.807714 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 01:46:34.807714 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 01:46:34.807714 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 01:46:34.807714 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.creator() Aug 30 01:46:34.807714 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 01:46:34.807714 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application created_value = creator( Aug 30 01:46:34.808773 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 30 01:46:34.808773 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 30 01:46:34.808773 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 30 01:46:34.808773 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 30 01:46:34.808773 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 30 01:46:34.808773 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 30 01:46:34.808773 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 30 01:46:34.808773 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Aug 30 01:46:34.808773 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 613/1217] 10.222.0.27 () {56 vars in 1111 bytes} [Tue Aug 30 01:46:34 2022] GET /identity/v3/projects/service => generated 88 bytes in 159 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 01:46:34.813697 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7304ea4e-37b3-43db-a841-59128e1a5d48 None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:46:34.914128 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7304ea4e-37b3-43db-a841-59128e1a5d48 None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:46:34.936681 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:46:34.948155 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c1f4a8b-7c1b-4e79-b77c-047304da365d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:46:34.948951 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c1f4a8b-7c1b-4e79-b77c-047304da365d None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:46:34.949211 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c1f4a8b-7c1b-4e79-b77c-047304da365d None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:46:34.949507 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c1f4a8b-7c1b-4e79-b77c-047304da365d None admin] PATH_INFO: `/v3/projects` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:46:34.950197 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c1f4a8b-7c1b-4e79-b77c-047304da365d None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81578) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:46:34.950501 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c1f4a8b-7c1b-4e79-b77c-047304da365d None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:46:34.951919 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c1f4a8b-7c1b-4e79-b77c-047304da365d None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:46:34.963691 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 605/1218] 10.222.0.27 () {56 vars in 1148 bytes} [Tue Aug 30 01:46:34 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:46:34.969836 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c02a8e7c-be15-49ca-9a62-386b17cfc657 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:46:35.071686 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c02a8e7c-be15-49ca-9a62-386b17cfc657 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:46:35.099581 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:46:35.105387 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4caac930-e029-47d2-a072-a7b7e5ebad63 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:46:35.106005 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4caac930-e029-47d2-a072-a7b7e5ebad63 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:46:35.106281 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4caac930-e029-47d2-a072-a7b7e5ebad63 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:46:35.106548 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4caac930-e029-47d2-a072-a7b7e5ebad63 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:46:35.107263 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4caac930-e029-47d2-a072-a7b7e5ebad63 None admin] RBAC: Adding query filter params (role.id=aa59b0c9009a480994d9df81a5f8d404, scope.project.id=4562f3951fbe465280073e60a11e44e2, user.id=379536ff06fd4f719899e5dde732e63d) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:46:35.107591 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4caac930-e029-47d2-a072-a7b7e5ebad63 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:46:35.109123 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4caac930-e029-47d2-a072-a7b7e5ebad63 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:46:35.114421 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 614/1219] 10.222.0.27 () {56 vars in 1374 bytes} [Tue Aug 30 01:46:34 2022] GET /identity/v3/role_assignments?user.id=379536ff06fd4f719899e5dde732e63d&scope.project.id=4562f3951fbe465280073e60a11e44e2&role.id=aa59b0c9009a480994d9df81a5f8d404 => generated 259 bytes in 145 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:46:37.007714 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48d16bee-5b66-44c4-bc18-852a651979cf None None] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:46:37.008219 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48d16bee-5b66-44c4-bc18-852a651979cf None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:46:37.008219 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48d16bee-5b66-44c4-bc18-852a651979cf None None] PATH_INFO: `/` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:46:37.009505 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 606/1220] 10.222.0.27 () {54 vars in 927 bytes} [Tue Aug 30 01:46:37 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 01:46:37.014770 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dc4af9a-360c-492e-809f-50bba02cc0a1 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:46:37.015227 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dc4af9a-360c-492e-809f-50bba02cc0a1 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:46:37.015318 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dc4af9a-360c-492e-809f-50bba02cc0a1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:46:37.033329 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-6dc4af9a-360c-492e-809f-50bba02cc0a1 None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 01:46:37.034548 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-6dc4af9a-360c-492e-809f-50bba02cc0a1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:46:37.101179 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 615/1221] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Aug 30 01:46:37 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 30 01:46:37.375447 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48d16bee-5b66-44c4-bc18-852a651979cf None None] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:46:37.481973 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48d16bee-5b66-44c4-bc18-852a651979cf None None] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:46:37.517561 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6234b7e3-8e3f-46b0-9a0b-d2de957ec127 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:46:37.518182 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6234b7e3-8e3f-46b0-9a0b-d2de957ec127 None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:46:37.518460 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6234b7e3-8e3f-46b0-9a0b-d2de957ec127 None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:46:37.518726 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6234b7e3-8e3f-46b0-9a0b-d2de957ec127 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:46:37.529215 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6234b7e3-8e3f-46b0-9a0b-d2de957ec127 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 30 01:46:37.533980 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application [None req-6234b7e3-8e3f-46b0-9a0b-d2de957ec127 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Aug 30 01:46:37.533980 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 01:46:37.533980 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 01:46:37.533980 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 01:46:37.533980 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 01:46:37.533980 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 01:46:37.533980 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 01:46:37.533980 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 01:46:37.533980 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 01:46:37.533980 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 01:46:37.533980 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 01:46:37.533980 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 01:46:37.533980 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 30 01:46:37.533980 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 30 01:46:37.533980 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 30 01:46:37.533980 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 30 01:46:37.535047 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 30 01:46:37.535047 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 30 01:46:37.535047 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 30 01:46:37.535047 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 30 01:46:37.535047 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 01:46:37.535047 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 01:46:37.535047 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 01:46:37.535047 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 01:46:37.535047 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 01:46:37.535047 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 01:46:37.535047 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 01:46:37.535047 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application with Lock( Aug 30 01:46:37.535047 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 01:46:37.535047 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._enter() Aug 30 01:46:37.535047 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 01:46:37.535047 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 01:46:37.536834 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 01:46:37.536834 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.creator() Aug 30 01:46:37.536834 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 01:46:37.536834 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application created_value = creator( Aug 30 01:46:37.536834 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 30 01:46:37.536834 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 30 01:46:37.536834 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 30 01:46:37.536834 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 30 01:46:37.536834 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 30 01:46:37.536834 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 30 01:46:37.536834 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Aug 30 01:46:37.536834 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application Aug 30 01:46:37.536834 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 607/1222] 10.222.0.27 () {56 vars in 1102 bytes} [Tue Aug 30 01:46:37 2022] GET /identity/v3/roles/service => generated 85 bytes in 160 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 01:46:37.543481 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6dc4af9a-360c-492e-809f-50bba02cc0a1 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:46:37.640124 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6dc4af9a-360c-492e-809f-50bba02cc0a1 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:46:37.663951 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:46:37.674134 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c96fc3bc-452f-44e2-8a8c-49157d9e27f7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:46:37.674730 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c96fc3bc-452f-44e2-8a8c-49157d9e27f7 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:46:37.675031 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c96fc3bc-452f-44e2-8a8c-49157d9e27f7 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:46:37.675303 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c96fc3bc-452f-44e2-8a8c-49157d9e27f7 None admin] PATH_INFO: `/v3/roles` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:46:37.676088 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c96fc3bc-452f-44e2-8a8c-49157d9e27f7 None admin] RBAC: Adding query filter params (name=service) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:46:37.676427 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c96fc3bc-452f-44e2-8a8c-49157d9e27f7 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:46:37.677806 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c96fc3bc-452f-44e2-8a8c-49157d9e27f7 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:46:37.685787 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 616/1223] 10.222.0.27 () {56 vars in 1103 bytes} [Tue Aug 30 01:46:37 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 144 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:46:37.691705 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6234b7e3-8e3f-46b0-9a0b-d2de957ec127 None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:46:37.736644 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:46:37.793296 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6234b7e3-8e3f-46b0-9a0b-d2de957ec127 None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:46:37.827414 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c71203aa-91cb-4425-b4fc-6dfa69e0d8b9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:46:37.828025 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c71203aa-91cb-4425-b4fc-6dfa69e0d8b9 None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:46:37.828297 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c71203aa-91cb-4425-b4fc-6dfa69e0d8b9 None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:46:37.828565 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c71203aa-91cb-4425-b4fc-6dfa69e0d8b9 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:46:37.839040 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c71203aa-91cb-4425-b4fc-6dfa69e0d8b9 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:46:37.840586 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c71203aa-91cb-4425-b4fc-6dfa69e0d8b9 None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:46:37.855320 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application [None req-c71203aa-91cb-4425-b4fc-6dfa69e0d8b9 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 01:46:37.855320 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 01:46:37.855320 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 01:46:37.855320 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 01:46:37.855320 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 01:46:37.855320 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 01:46:37.855320 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 01:46:37.855320 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 01:46:37.855320 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 01:46:37.855320 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 01:46:37.855320 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 01:46:37.855320 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 01:46:37.855320 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 30 01:46:37.855320 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 30 01:46:37.855320 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 30 01:46:37.855320 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 30 01:46:37.856377 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 01:46:37.856377 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 01:46:37.856377 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 01:46:37.856377 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 01:46:37.856377 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 01:46:37.856377 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 01:46:37.856377 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 01:46:37.856377 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application with Lock( Aug 30 01:46:37.856377 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 01:46:37.856377 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._enter() Aug 30 01:46:37.856377 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 01:46:37.856377 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 01:46:37.856377 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 01:46:37.856377 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.creator() Aug 30 01:46:37.856377 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 01:46:37.856377 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application created_value = creator( Aug 30 01:46:37.856377 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 30 01:46:37.857343 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 30 01:46:37.857343 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 01:46:37.857343 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application Aug 30 01:46:37.857343 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 608/1224] 10.222.0.27 () {56 vars in 1108 bytes} [Tue Aug 30 01:46:37 2022] GET /identity/v3/domains/Default => generated 87 bytes in 165 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 01:46:37.862442 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c96fc3bc-452f-44e2-8a8c-49157d9e27f7 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:46:37.961783 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c96fc3bc-452f-44e2-8a8c-49157d9e27f7 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:46:37.996107 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d16ab534-50b8-48ee-aa4d-b92c0aae28f8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:46:37.996730 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d16ab534-50b8-48ee-aa4d-b92c0aae28f8 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:46:37.997013 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d16ab534-50b8-48ee-aa4d-b92c0aae28f8 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:46:37.997287 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d16ab534-50b8-48ee-aa4d-b92c0aae28f8 None admin] PATH_INFO: `/v3/domains` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:46:37.998027 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d16ab534-50b8-48ee-aa4d-b92c0aae28f8 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:46:37.998332 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d16ab534-50b8-48ee-aa4d-b92c0aae28f8 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:46:37.999699 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d16ab534-50b8-48ee-aa4d-b92c0aae28f8 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:46:38.015263 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 617/1225] 10.222.0.27 () {56 vars in 1109 bytes} [Tue Aug 30 01:46:37 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:46:38.021025 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c71203aa-91cb-4425-b4fc-6dfa69e0d8b9 None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:46:38.114176 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:46:38.120685 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c71203aa-91cb-4425-b4fc-6dfa69e0d8b9 None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:46:38.153730 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c97e64c5-b513-44e3-a4ee-f9d3ad37ea59 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:46:38.154345 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c97e64c5-b513-44e3-a4ee-f9d3ad37ea59 None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:46:38.154624 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c97e64c5-b513-44e3-a4ee-f9d3ad37ea59 None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:46:38.154893 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c97e64c5-b513-44e3-a4ee-f9d3ad37ea59 None admin] PATH_INFO: `/v3/users/placement` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:46:38.162229 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c97e64c5-b513-44e3-a4ee-f9d3ad37ea59 None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:46:38.163656 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c97e64c5-b513-44e3-a4ee-f9d3ad37ea59 None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:46:38.174514 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application [None req-c97e64c5-b513-44e3-a4ee-f9d3ad37ea59 None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Aug 30 01:46:38.174514 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 01:46:38.174514 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 01:46:38.174514 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 01:46:38.174514 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 01:46:38.174514 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 01:46:38.174514 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 01:46:38.174514 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 01:46:38.174514 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 01:46:38.174514 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 01:46:38.174514 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 01:46:38.174514 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 01:46:38.174514 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 30 01:46:38.174514 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 30 01:46:38.174514 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 30 01:46:38.174514 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 30 01:46:38.185438 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 01:46:38.185438 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 01:46:38.185438 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 30 01:46:38.185438 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 30 01:46:38.185438 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 30 01:46:38.185438 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 30 01:46:38.185438 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 01:46:38.185438 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 01:46:38.185438 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 01:46:38.185438 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 01:46:38.185438 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 01:46:38.185438 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application with Lock( Aug 30 01:46:38.185438 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 01:46:38.185438 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._enter() Aug 30 01:46:38.185438 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 01:46:38.185438 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 01:46:38.185438 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 01:46:38.185438 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.creator() Aug 30 01:46:38.186648 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 01:46:38.186648 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application created_value = creator( Aug 30 01:46:38.186648 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 30 01:46:38.186648 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 30 01:46:38.186648 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 30 01:46:38.186648 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 30 01:46:38.186648 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 30 01:46:38.186648 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 30 01:46:38.186648 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Aug 30 01:46:38.186648 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application Aug 30 01:46:38.186648 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 609/1226] 10.222.0.27 () {56 vars in 1108 bytes} [Tue Aug 30 01:46:38 2022] GET /identity/v3/users/placement => generated 87 bytes in 155 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 01:46:38.187418 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d16ab534-50b8-48ee-aa4d-b92c0aae28f8 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:46:38.233964 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:46:38.286901 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d16ab534-50b8-48ee-aa4d-b92c0aae28f8 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:46:38.321336 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ce0053f-4898-4bed-ba6e-d84e6a5fcb0b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:46:38.322252 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ce0053f-4898-4bed-ba6e-d84e6a5fcb0b None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:46:38.322741 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ce0053f-4898-4bed-ba6e-d84e6a5fcb0b None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:46:38.323205 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ce0053f-4898-4bed-ba6e-d84e6a5fcb0b None admin] PATH_INFO: `/v3/users` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:46:38.323930 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ce0053f-4898-4bed-ba6e-d84e6a5fcb0b None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:46:38.324233 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ce0053f-4898-4bed-ba6e-d84e6a5fcb0b None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:46:38.326064 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ce0053f-4898-4bed-ba6e-d84e6a5fcb0b None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:46:38.345937 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 618/1227] 10.222.0.27 () {56 vars in 1143 bytes} [Tue Aug 30 01:46:38 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:46:38.352034 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c97e64c5-b513-44e3-a4ee-f9d3ad37ea59 None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:46:38.452454 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c97e64c5-b513-44e3-a4ee-f9d3ad37ea59 None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:46:38.485833 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4c96e7c-8f69-4f67-926b-7ca857764148 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:46:38.486383 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4c96e7c-8f69-4f67-926b-7ca857764148 None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:46:38.486661 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4c96e7c-8f69-4f67-926b-7ca857764148 None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:46:38.486929 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4c96e7c-8f69-4f67-926b-7ca857764148 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:46:38.497356 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4c96e7c-8f69-4f67-926b-7ca857764148 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:46:38.499759 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4c96e7c-8f69-4f67-926b-7ca857764148 None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:46:38.512785 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application [None req-a4c96e7c-8f69-4f67-926b-7ca857764148 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 01:46:38.512785 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 01:46:38.512785 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 01:46:38.512785 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 01:46:38.512785 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 01:46:38.512785 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 01:46:38.512785 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 01:46:38.512785 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 01:46:38.512785 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 01:46:38.512785 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 01:46:38.512785 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 01:46:38.512785 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 01:46:38.512785 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 30 01:46:38.512785 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 30 01:46:38.512785 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 30 01:46:38.512785 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 30 01:46:38.513954 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 01:46:38.513954 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 01:46:38.513954 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 01:46:38.513954 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 01:46:38.513954 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 01:46:38.513954 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 01:46:38.513954 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 01:46:38.513954 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application with Lock( Aug 30 01:46:38.513954 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 01:46:38.513954 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._enter() Aug 30 01:46:38.513954 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 01:46:38.513954 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 01:46:38.513954 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 01:46:38.513954 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.creator() Aug 30 01:46:38.513954 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 01:46:38.513954 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application created_value = creator( Aug 30 01:46:38.513954 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 30 01:46:38.515254 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 30 01:46:38.515254 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 01:46:38.515254 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application Aug 30 01:46:38.515254 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 610/1228] 10.222.0.27 () {56 vars in 1108 bytes} [Tue Aug 30 01:46:38 2022] GET /identity/v3/domains/Default => generated 87 bytes in 162 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 01:46:38.519692 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ce0053f-4898-4bed-ba6e-d84e6a5fcb0b None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:46:38.623081 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ce0053f-4898-4bed-ba6e-d84e6a5fcb0b None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:46:38.625613 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:46:38.657445 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42357e0c-c429-4a19-a49b-bdfc1124c2e9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:46:38.658073 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42357e0c-c429-4a19-a49b-bdfc1124c2e9 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:46:38.658355 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42357e0c-c429-4a19-a49b-bdfc1124c2e9 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:46:38.658634 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42357e0c-c429-4a19-a49b-bdfc1124c2e9 None admin] PATH_INFO: `/v3/domains` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:46:38.659275 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42357e0c-c429-4a19-a49b-bdfc1124c2e9 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:46:38.659578 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42357e0c-c429-4a19-a49b-bdfc1124c2e9 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:46:38.660947 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42357e0c-c429-4a19-a49b-bdfc1124c2e9 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:46:38.672291 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 619/1229] 10.222.0.27 () {56 vars in 1109 bytes} [Tue Aug 30 01:46:38 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:46:38.678490 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4c96e7c-8f69-4f67-926b-7ca857764148 None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:46:38.688207 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:46:38.782731 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4c96e7c-8f69-4f67-926b-7ca857764148 None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:46:38.815799 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-824cc641-8b75-4753-8a12-541a9a1a6504 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:46:38.816414 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-824cc641-8b75-4753-8a12-541a9a1a6504 None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:46:38.816693 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-824cc641-8b75-4753-8a12-541a9a1a6504 None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:46:38.817104 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-824cc641-8b75-4753-8a12-541a9a1a6504 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:46:38.823617 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-824cc641-8b75-4753-8a12-541a9a1a6504 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:46:38.825605 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-824cc641-8b75-4753-8a12-541a9a1a6504 None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:46:38.836659 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application [None req-824cc641-8b75-4753-8a12-541a9a1a6504 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 30 01:46:38.836659 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 01:46:38.836659 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 01:46:38.836659 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 01:46:38.836659 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 01:46:38.836659 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 01:46:38.836659 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 01:46:38.836659 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 01:46:38.836659 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 01:46:38.836659 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 01:46:38.836659 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 01:46:38.836659 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 01:46:38.836659 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 30 01:46:38.836659 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 30 01:46:38.836659 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 30 01:46:38.837766 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 30 01:46:38.837766 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 01:46:38.837766 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 01:46:38.837766 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 01:46:38.837766 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 01:46:38.837766 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 01:46:38.837766 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 01:46:38.837766 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 01:46:38.837766 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application with Lock( Aug 30 01:46:38.837766 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 01:46:38.837766 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._enter() Aug 30 01:46:38.837766 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 01:46:38.837766 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 01:46:38.837766 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 01:46:38.837766 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.creator() Aug 30 01:46:38.837766 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 01:46:38.837766 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application created_value = creator( Aug 30 01:46:38.838705 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 30 01:46:38.838705 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 30 01:46:38.838705 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 30 01:46:38.838705 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 30 01:46:38.838705 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 30 01:46:38.838705 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 30 01:46:38.838705 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 30 01:46:38.838705 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application Aug 30 01:46:38.838705 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 611/1230] 10.222.0.27 () {56 vars in 1111 bytes} [Tue Aug 30 01:46:38 2022] GET /identity/v3/projects/service => generated 88 bytes in 160 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 01:46:38.843743 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42357e0c-c429-4a19-a49b-bdfc1124c2e9 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:46:38.945519 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42357e0c-c429-4a19-a49b-bdfc1124c2e9 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:46:38.979185 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08c2c881-decb-4044-931e-65afa2c3ba60 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:46:38.979784 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08c2c881-decb-4044-931e-65afa2c3ba60 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:46:38.980062 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08c2c881-decb-4044-931e-65afa2c3ba60 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:46:38.980333 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08c2c881-decb-4044-931e-65afa2c3ba60 None admin] PATH_INFO: `/v3/projects` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:46:38.981016 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08c2c881-decb-4044-931e-65afa2c3ba60 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:46:38.981320 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08c2c881-decb-4044-931e-65afa2c3ba60 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:46:38.982772 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08c2c881-decb-4044-931e-65afa2c3ba60 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:46:38.995034 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 620/1231] 10.222.0.27 () {56 vars in 1148 bytes} [Tue Aug 30 01:46:38 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 30 01:46:39.001285 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-824cc641-8b75-4753-8a12-541a9a1a6504 None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:46:39.066861 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:46:39.101135 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-824cc641-8b75-4753-8a12-541a9a1a6504 None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:46:39.134695 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7882dc4-d9ad-4e6d-9e88-4febf6f1cc0f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:46:39.135339 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7882dc4-d9ad-4e6d-9e88-4febf6f1cc0f None admin] REQUEST_METHOD: `PUT` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:46:39.135613 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7882dc4-d9ad-4e6d-9e88-4febf6f1cc0f None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:46:39.135879 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7882dc4-d9ad-4e6d-9e88-4febf6f1cc0f None admin] PATH_INFO: `/v3/projects/4562f3951fbe465280073e60a11e44e2/users/379536ff06fd4f719899e5dde732e63d/roles/aa59b0c9009a480994d9df81a5f8d404` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:46:39.150091 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7882dc4-d9ad-4e6d-9e88-4febf6f1cc0f None admin] RBAC: Authorizing `identity:create_grant(project_id=4562f3951fbe465280073e60a11e44e2, user_id=379536ff06fd4f719899e5dde732e63d, role_id=aa59b0c9009a480994d9df81a5f8d404)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:46:39.151728 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7882dc4-d9ad-4e6d-9e88-4febf6f1cc0f None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:46:39.218798 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 612/1232] 10.222.0.27 () {58 vars in 1439 bytes} [Tue Aug 30 01:46:39 2022] PUT /identity/v3/projects/4562f3951fbe465280073e60a11e44e2/users/379536ff06fd4f719899e5dde732e63d/roles/aa59b0c9009a480994d9df81a5f8d404 => generated 0 bytes in 218 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 01:46:41.186155 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63a22982-5cad-40fe-baec-a2ca78065780 None None] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:46:41.186741 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63a22982-5cad-40fe-baec-a2ca78065780 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:46:41.186741 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63a22982-5cad-40fe-baec-a2ca78065780 None None] PATH_INFO: `/` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:46:41.188042 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 621/1233] 10.222.0.27 () {54 vars in 927 bytes} [Tue Aug 30 01:46:41 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 01:46:41.194386 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76d88587-ca90-46ac-9a50-3ca1bf188a14 None None] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:46:41.194832 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76d88587-ca90-46ac-9a50-3ca1bf188a14 None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:46:41.194943 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76d88587-ca90-46ac-9a50-3ca1bf188a14 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:46:41.214237 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.auth.core [None req-76d88587-ca90-46ac-9a50-3ca1bf188a14 None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81578) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 01:46:41.215724 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-76d88587-ca90-46ac-9a50-3ca1bf188a14 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:46:41.295274 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 613/1234] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Aug 30 01:46:41 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 30 01:46:41.565773 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63a22982-5cad-40fe-baec-a2ca78065780 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:46:41.593424 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:46:41.676706 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63a22982-5cad-40fe-baec-a2ca78065780 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:46:41.714785 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-637151b3-ce24-4736-8da0-94a23eb27f27 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:46:41.715493 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-637151b3-ce24-4736-8da0-94a23eb27f27 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:46:41.715833 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-637151b3-ce24-4736-8da0-94a23eb27f27 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:46:41.716219 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-637151b3-ce24-4736-8da0-94a23eb27f27 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:46:41.728328 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-637151b3-ce24-4736-8da0-94a23eb27f27 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 30 01:46:41.733226 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application [None req-637151b3-ce24-4736-8da0-94a23eb27f27 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Aug 30 01:46:41.733226 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 01:46:41.733226 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 01:46:41.733226 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 01:46:41.733226 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 01:46:41.733226 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 01:46:41.733226 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 01:46:41.733226 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 01:46:41.733226 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 01:46:41.733226 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 01:46:41.733226 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 01:46:41.733226 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 01:46:41.733226 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 30 01:46:41.733226 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 30 01:46:41.733226 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 30 01:46:41.733226 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 30 01:46:41.734427 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 30 01:46:41.734427 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 30 01:46:41.734427 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 30 01:46:41.734427 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 30 01:46:41.734427 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 01:46:41.734427 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 01:46:41.734427 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 01:46:41.734427 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 01:46:41.734427 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 01:46:41.734427 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 01:46:41.734427 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 01:46:41.734427 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application with Lock( Aug 30 01:46:41.734427 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 01:46:41.734427 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._enter() Aug 30 01:46:41.734427 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 01:46:41.734427 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 01:46:41.735279 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 01:46:41.735279 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.creator() Aug 30 01:46:41.735279 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 01:46:41.735279 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application created_value = creator( Aug 30 01:46:41.735279 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 30 01:46:41.735279 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 30 01:46:41.735279 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 30 01:46:41.735279 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 30 01:46:41.735279 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 30 01:46:41.735279 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 30 01:46:41.735279 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Aug 30 01:46:41.735279 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Aug 30 01:46:41.735279 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 622/1235] 10.222.0.27 () {56 vars in 1102 bytes} [Tue Aug 30 01:46:41 2022] GET /identity/v3/roles/service => generated 85 bytes in 169 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 01:46:41.741238 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76d88587-ca90-46ac-9a50-3ca1bf188a14 None None] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:46:41.786483 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:46:41.836764 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76d88587-ca90-46ac-9a50-3ca1bf188a14 None None] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:46:41.870204 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60f2e77b-ff21-4d10-acb6-315d00066be1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:46:41.870804 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60f2e77b-ff21-4d10-acb6-315d00066be1 None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:46:41.871081 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60f2e77b-ff21-4d10-acb6-315d00066be1 None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:46:41.871350 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60f2e77b-ff21-4d10-acb6-315d00066be1 None admin] PATH_INFO: `/v3/roles` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:46:41.872008 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60f2e77b-ff21-4d10-acb6-315d00066be1 None admin] RBAC: Adding query filter params (name=service) {{(pid=81578) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:46:41.872308 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60f2e77b-ff21-4d10-acb6-315d00066be1 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:46:41.873682 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60f2e77b-ff21-4d10-acb6-315d00066be1 None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:46:41.881437 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 614/1236] 10.222.0.27 () {56 vars in 1103 bytes} [Tue Aug 30 01:46:41 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 141 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:46:41.887318 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-637151b3-ce24-4736-8da0-94a23eb27f27 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:46:41.970562 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:46:41.986662 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-637151b3-ce24-4736-8da0-94a23eb27f27 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:46:42.026759 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e01bc0c-80bf-49d5-bbb7-f9765f0d0311 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:46:42.027399 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e01bc0c-80bf-49d5-bbb7-f9765f0d0311 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:46:42.027680 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e01bc0c-80bf-49d5-bbb7-f9765f0d0311 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:46:42.027950 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e01bc0c-80bf-49d5-bbb7-f9765f0d0311 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:46:42.042140 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e01bc0c-80bf-49d5-bbb7-f9765f0d0311 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:46:42.043604 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e01bc0c-80bf-49d5-bbb7-f9765f0d0311 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:46:42.058093 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application [None req-7e01bc0c-80bf-49d5-bbb7-f9765f0d0311 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 01:46:42.058093 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 01:46:42.058093 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 01:46:42.058093 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 01:46:42.058093 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 01:46:42.058093 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 01:46:42.058093 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 01:46:42.058093 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 01:46:42.058093 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 01:46:42.058093 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 01:46:42.058093 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 01:46:42.058093 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 01:46:42.058093 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 30 01:46:42.058093 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 30 01:46:42.058093 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 30 01:46:42.058093 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 30 01:46:42.059246 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 01:46:42.059246 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 01:46:42.059246 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 01:46:42.059246 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 01:46:42.059246 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 01:46:42.059246 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 01:46:42.059246 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 01:46:42.059246 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application with Lock( Aug 30 01:46:42.059246 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 01:46:42.059246 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._enter() Aug 30 01:46:42.059246 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 01:46:42.059246 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 01:46:42.059246 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 01:46:42.059246 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.creator() Aug 30 01:46:42.059246 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 01:46:42.059246 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application created_value = creator( Aug 30 01:46:42.059246 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 30 01:46:42.060350 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 30 01:46:42.060350 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 01:46:42.060350 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Aug 30 01:46:42.060350 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 623/1237] 10.222.0.27 () {56 vars in 1108 bytes} [Tue Aug 30 01:46:41 2022] GET /identity/v3/domains/Default => generated 87 bytes in 172 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 01:46:42.065338 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60f2e77b-ff21-4d10-acb6-315d00066be1 None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:46:42.164994 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60f2e77b-ff21-4d10-acb6-315d00066be1 None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:46:42.173435 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:46:42.200611 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a61a364-1e4f-4337-9d54-5479ee05dda4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:46:42.201515 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a61a364-1e4f-4337-9d54-5479ee05dda4 None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:46:42.201801 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a61a364-1e4f-4337-9d54-5479ee05dda4 None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:46:42.202075 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a61a364-1e4f-4337-9d54-5479ee05dda4 None admin] PATH_INFO: `/v3/domains` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:46:42.202720 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a61a364-1e4f-4337-9d54-5479ee05dda4 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81578) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:46:42.203023 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a61a364-1e4f-4337-9d54-5479ee05dda4 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:46:42.204387 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a61a364-1e4f-4337-9d54-5479ee05dda4 None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:46:42.218924 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 615/1238] 10.222.0.27 () {56 vars in 1109 bytes} [Tue Aug 30 01:46:42 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:46:42.224783 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e01bc0c-80bf-49d5-bbb7-f9765f0d0311 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:46:42.325599 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e01bc0c-80bf-49d5-bbb7-f9765f0d0311 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:46:42.369027 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43ad3182-5155-4c25-a766-720a56306f2c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:46:42.369682 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43ad3182-5155-4c25-a766-720a56306f2c None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:46:42.369961 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43ad3182-5155-4c25-a766-720a56306f2c None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:46:42.370229 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43ad3182-5155-4c25-a766-720a56306f2c None admin] PATH_INFO: `/v3/users/placement` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:46:42.373023 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:46:42.378432 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43ad3182-5155-4c25-a766-720a56306f2c None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:46:42.380060 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43ad3182-5155-4c25-a766-720a56306f2c None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:46:42.391999 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application [None req-43ad3182-5155-4c25-a766-720a56306f2c None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Aug 30 01:46:42.391999 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 01:46:42.391999 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 01:46:42.391999 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 01:46:42.391999 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 01:46:42.391999 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 01:46:42.391999 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 01:46:42.391999 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 01:46:42.391999 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 01:46:42.391999 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 01:46:42.391999 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 01:46:42.391999 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 01:46:42.391999 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 30 01:46:42.391999 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 30 01:46:42.391999 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 30 01:46:42.391999 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 30 01:46:42.392992 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 01:46:42.392992 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 01:46:42.392992 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 30 01:46:42.392992 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 30 01:46:42.392992 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 30 01:46:42.392992 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 30 01:46:42.392992 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 01:46:42.392992 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 01:46:42.392992 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 01:46:42.392992 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 01:46:42.392992 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 01:46:42.392992 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application with Lock( Aug 30 01:46:42.392992 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 01:46:42.392992 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._enter() Aug 30 01:46:42.392992 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 01:46:42.392992 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 01:46:42.392992 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 01:46:42.392992 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.creator() Aug 30 01:46:42.394109 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 01:46:42.394109 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application created_value = creator( Aug 30 01:46:42.394109 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 30 01:46:42.394109 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 30 01:46:42.394109 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 30 01:46:42.394109 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 30 01:46:42.394109 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 30 01:46:42.394109 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 30 01:46:42.394109 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Aug 30 01:46:42.394109 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Aug 30 01:46:42.394109 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 624/1239] 10.222.0.27 () {56 vars in 1108 bytes} [Tue Aug 30 01:46:42 2022] GET /identity/v3/users/placement => generated 87 bytes in 169 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 01:46:42.400347 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a61a364-1e4f-4337-9d54-5479ee05dda4 None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:46:42.499179 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a61a364-1e4f-4337-9d54-5479ee05dda4 None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:46:42.535947 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa1489f9-3ead-458f-8975-160a31dd2e51 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:46:42.536729 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa1489f9-3ead-458f-8975-160a31dd2e51 None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:46:42.537012 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa1489f9-3ead-458f-8975-160a31dd2e51 None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:46:42.544300 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa1489f9-3ead-458f-8975-160a31dd2e51 None admin] PATH_INFO: `/v3/users` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:46:42.544300 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa1489f9-3ead-458f-8975-160a31dd2e51 None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=81578) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:46:42.544300 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa1489f9-3ead-458f-8975-160a31dd2e51 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:46:42.544300 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa1489f9-3ead-458f-8975-160a31dd2e51 None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:46:42.557443 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 616/1240] 10.222.0.27 () {56 vars in 1143 bytes} [Tue Aug 30 01:46:42 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:46:42.563797 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43ad3182-5155-4c25-a766-720a56306f2c None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:46:42.664569 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43ad3182-5155-4c25-a766-720a56306f2c None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:46:42.697699 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1671890-f898-4753-925b-e1334fcc0f32 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:46:42.698289 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1671890-f898-4753-925b-e1334fcc0f32 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:46:42.698565 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1671890-f898-4753-925b-e1334fcc0f32 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:46:42.698830 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1671890-f898-4753-925b-e1334fcc0f32 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:46:42.761569 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:46:42.768017 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1671890-f898-4753-925b-e1334fcc0f32 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:46:42.769485 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1671890-f898-4753-925b-e1334fcc0f32 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:46:42.868113 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 625/1241] 10.222.0.27 () {58 vars in 1282 bytes} [Tue Aug 30 01:46:42 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 305 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Aug 30 01:46:42.873961 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa1489f9-3ead-458f-8975-160a31dd2e51 None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:46:42.901296 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:46:42.971054 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa1489f9-3ead-458f-8975-160a31dd2e51 None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:46:43.006463 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c4b93b3-009a-46ee-908a-c018d5ca67b7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:46:43.007083 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c4b93b3-009a-46ee-908a-c018d5ca67b7 None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:46:43.007363 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c4b93b3-009a-46ee-908a-c018d5ca67b7 None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:46:43.007630 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c4b93b3-009a-46ee-908a-c018d5ca67b7 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:46:43.017524 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c4b93b3-009a-46ee-908a-c018d5ca67b7 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:46:43.018869 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c4b93b3-009a-46ee-908a-c018d5ca67b7 None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:46:43.034033 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application [None req-7c4b93b3-009a-46ee-908a-c018d5ca67b7 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 01:46:43.034033 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 01:46:43.034033 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 01:46:43.034033 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 01:46:43.034033 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 01:46:43.034033 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 01:46:43.034033 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 01:46:43.034033 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 01:46:43.034033 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 01:46:43.034033 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 01:46:43.034033 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 01:46:43.034033 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 01:46:43.034033 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 30 01:46:43.034033 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 30 01:46:43.034033 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 30 01:46:43.034033 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 30 01:46:43.035137 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 01:46:43.035137 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 01:46:43.035137 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 01:46:43.035137 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 01:46:43.035137 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 01:46:43.035137 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 01:46:43.035137 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 01:46:43.035137 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application with Lock( Aug 30 01:46:43.035137 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 01:46:43.035137 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._enter() Aug 30 01:46:43.035137 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 01:46:43.035137 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 01:46:43.035137 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 01:46:43.035137 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.creator() Aug 30 01:46:43.035137 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 01:46:43.035137 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application created_value = creator( Aug 30 01:46:43.035137 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 30 01:46:43.036942 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 30 01:46:43.036942 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 01:46:43.036942 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application Aug 30 01:46:43.036942 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 617/1242] 10.222.0.27 () {56 vars in 1108 bytes} [Tue Aug 30 01:46:42 2022] GET /identity/v3/domains/Default => generated 87 bytes in 162 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 01:46:43.041392 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1671890-f898-4753-925b-e1334fcc0f32 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:46:43.134403 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:46:43.140027 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1671890-f898-4753-925b-e1334fcc0f32 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:46:43.173783 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4033161-04a7-48d0-99db-54e341c6f021 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:46:43.174377 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4033161-04a7-48d0-99db-54e341c6f021 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:46:43.174651 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4033161-04a7-48d0-99db-54e341c6f021 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:46:43.174914 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4033161-04a7-48d0-99db-54e341c6f021 None admin] PATH_INFO: `/v3/domains` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:46:43.175545 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4033161-04a7-48d0-99db-54e341c6f021 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:46:43.175842 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4033161-04a7-48d0-99db-54e341c6f021 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:46:43.177895 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4033161-04a7-48d0-99db-54e341c6f021 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:46:43.189238 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 626/1243] 10.222.0.27 () {56 vars in 1109 bytes} [Tue Aug 30 01:46:43 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:46:43.195099 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c4b93b3-009a-46ee-908a-c018d5ca67b7 None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:46:43.273418 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:46:43.294543 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c4b93b3-009a-46ee-908a-c018d5ca67b7 None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:46:43.327804 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74473628-1082-412a-a62f-4068e5a50626 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:46:43.328435 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74473628-1082-412a-a62f-4068e5a50626 None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:46:43.328696 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74473628-1082-412a-a62f-4068e5a50626 None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:46:43.328960 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74473628-1082-412a-a62f-4068e5a50626 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:46:43.335682 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74473628-1082-412a-a62f-4068e5a50626 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:46:43.337099 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74473628-1082-412a-a62f-4068e5a50626 None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:46:43.348735 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application [None req-74473628-1082-412a-a62f-4068e5a50626 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 30 01:46:43.348735 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 01:46:43.348735 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 01:46:43.348735 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 01:46:43.348735 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 01:46:43.348735 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 01:46:43.348735 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 01:46:43.348735 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 01:46:43.348735 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 01:46:43.348735 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 01:46:43.348735 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 01:46:43.348735 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 01:46:43.348735 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 30 01:46:43.348735 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 30 01:46:43.348735 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 30 01:46:43.351560 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 30 01:46:43.351560 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 01:46:43.351560 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 01:46:43.351560 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 01:46:43.351560 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 01:46:43.351560 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 01:46:43.351560 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 01:46:43.351560 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 01:46:43.351560 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application with Lock( Aug 30 01:46:43.351560 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 01:46:43.351560 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._enter() Aug 30 01:46:43.351560 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 01:46:43.351560 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 01:46:43.351560 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 01:46:43.351560 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.creator() Aug 30 01:46:43.351560 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 01:46:43.351560 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application created_value = creator( Aug 30 01:46:43.352449 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 30 01:46:43.352449 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 30 01:46:43.352449 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 30 01:46:43.352449 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 30 01:46:43.352449 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 30 01:46:43.352449 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 30 01:46:43.352449 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 30 01:46:43.352449 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application Aug 30 01:46:43.352449 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 618/1244] 10.222.0.27 () {56 vars in 1111 bytes} [Tue Aug 30 01:46:43 2022] GET /identity/v3/projects/service => generated 88 bytes in 155 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 01:46:43.355966 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4033161-04a7-48d0-99db-54e341c6f021 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:46:43.455579 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4033161-04a7-48d0-99db-54e341c6f021 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:46:43.489242 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af9e3c58-df15-4fe9-934d-d116cfedf92c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:46:43.490010 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af9e3c58-df15-4fe9-934d-d116cfedf92c None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:46:43.490288 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af9e3c58-df15-4fe9-934d-d116cfedf92c None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:46:43.490554 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af9e3c58-df15-4fe9-934d-d116cfedf92c None admin] PATH_INFO: `/v3/projects` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:46:43.491237 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af9e3c58-df15-4fe9-934d-d116cfedf92c None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:46:43.491595 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af9e3c58-df15-4fe9-934d-d116cfedf92c None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:46:43.493011 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af9e3c58-df15-4fe9-934d-d116cfedf92c None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:46:43.504737 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 627/1245] 10.222.0.27 () {56 vars in 1148 bytes} [Tue Aug 30 01:46:43 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:46:43.510941 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74473628-1082-412a-a62f-4068e5a50626 None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:46:43.612249 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74473628-1082-412a-a62f-4068e5a50626 None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:46:43.646844 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bcbd75e-6933-4ccd-bf0c-3764642c9a18 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:46:43.647732 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bcbd75e-6933-4ccd-bf0c-3764642c9a18 None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:46:43.648018 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bcbd75e-6933-4ccd-bf0c-3764642c9a18 None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:46:43.648293 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bcbd75e-6933-4ccd-bf0c-3764642c9a18 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:46:43.649017 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9bcbd75e-6933-4ccd-bf0c-3764642c9a18 None admin] RBAC: Adding query filter params (role.id=aa59b0c9009a480994d9df81a5f8d404, scope.project.id=4562f3951fbe465280073e60a11e44e2, user.id=379536ff06fd4f719899e5dde732e63d) {{(pid=81578) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:46:43.649332 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9bcbd75e-6933-4ccd-bf0c-3764642c9a18 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:46:43.650789 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9bcbd75e-6933-4ccd-bf0c-3764642c9a18 None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:46:43.656454 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 619/1246] 10.222.0.27 () {56 vars in 1374 bytes} [Tue Aug 30 01:46:43 2022] GET /identity/v3/role_assignments?user.id=379536ff06fd4f719899e5dde732e63d&scope.project.id=4562f3951fbe465280073e60a11e44e2&role.id=aa59b0c9009a480994d9df81a5f8d404 => generated 608 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:46:45.636043 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cacb6f8-6646-4b9a-b995-32d9f61513ed None None] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:46:45.636598 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cacb6f8-6646-4b9a-b995-32d9f61513ed None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:46:45.636598 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cacb6f8-6646-4b9a-b995-32d9f61513ed None None] PATH_INFO: `/` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:46:45.638076 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 628/1247] 10.222.0.27 () {54 vars in 927 bytes} [Tue Aug 30 01:46:45 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 01:46:45.643076 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68bc43b5-65af-409b-b89e-c8acfd75ac8f None None] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:46:45.643511 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68bc43b5-65af-409b-b89e-c8acfd75ac8f None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:46:45.643614 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68bc43b5-65af-409b-b89e-c8acfd75ac8f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:46:45.650807 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:46:45.662934 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.auth.core [None req-68bc43b5-65af-409b-b89e-c8acfd75ac8f None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81578) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 01:46:45.664119 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-68bc43b5-65af-409b-b89e-c8acfd75ac8f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:46:45.730113 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 620/1248] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Aug 30 01:46:45 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 30 01:46:46.006309 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cacb6f8-6646-4b9a-b995-32d9f61513ed None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:46:46.024678 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:46:46.110189 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cacb6f8-6646-4b9a-b995-32d9f61513ed None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:46:46.144129 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1ca91b3-5e42-4a7c-a123-3fa44e5a2ec4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:46:46.145162 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1ca91b3-5e42-4a7c-a123-3fa44e5a2ec4 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:46:46.145442 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1ca91b3-5e42-4a7c-a123-3fa44e5a2ec4 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:46:46.145741 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1ca91b3-5e42-4a7c-a123-3fa44e5a2ec4 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:46:46.155662 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1ca91b3-5e42-4a7c-a123-3fa44e5a2ec4 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 30 01:46:46.160929 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application [None req-f1ca91b3-5e42-4a7c-a123-3fa44e5a2ec4 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 30 01:46:46.160929 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 01:46:46.160929 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 01:46:46.160929 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 01:46:46.160929 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 01:46:46.160929 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 01:46:46.160929 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 01:46:46.160929 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 01:46:46.160929 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 01:46:46.160929 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 01:46:46.160929 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 01:46:46.160929 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 01:46:46.160929 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 30 01:46:46.160929 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 30 01:46:46.160929 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 30 01:46:46.160929 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 30 01:46:46.165535 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 30 01:46:46.165535 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 30 01:46:46.165535 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 30 01:46:46.165535 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 30 01:46:46.165535 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 01:46:46.165535 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 01:46:46.165535 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 01:46:46.165535 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 01:46:46.165535 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 01:46:46.165535 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 01:46:46.165535 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 01:46:46.165535 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application with Lock( Aug 30 01:46:46.165535 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 01:46:46.165535 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._enter() Aug 30 01:46:46.165535 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 01:46:46.165535 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 01:46:46.166682 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 01:46:46.166682 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.creator() Aug 30 01:46:46.166682 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 01:46:46.166682 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application created_value = creator( Aug 30 01:46:46.166682 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 30 01:46:46.166682 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 30 01:46:46.166682 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 30 01:46:46.166682 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 30 01:46:46.166682 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 30 01:46:46.166682 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 30 01:46:46.166682 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 30 01:46:46.166682 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Aug 30 01:46:46.166682 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 629/1249] 10.222.0.27 () {56 vars in 1096 bytes} [Tue Aug 30 01:46:46 2022] GET /identity/v3/roles/admin => generated 83 bytes in 156 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 01:46:46.168080 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68bc43b5-65af-409b-b89e-c8acfd75ac8f None None] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:46:46.270803 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68bc43b5-65af-409b-b89e-c8acfd75ac8f None None] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:46:46.299519 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:46:46.304932 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c06ed51-0ea6-4321-be85-3a38d699267a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:46:46.305554 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c06ed51-0ea6-4321-be85-3a38d699267a None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:46:46.305831 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c06ed51-0ea6-4321-be85-3a38d699267a None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:46:46.306124 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c06ed51-0ea6-4321-be85-3a38d699267a None admin] PATH_INFO: `/v3/roles` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:46:46.306781 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c06ed51-0ea6-4321-be85-3a38d699267a None admin] RBAC: Adding query filter params (name=admin) {{(pid=81578) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:46:46.307079 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c06ed51-0ea6-4321-be85-3a38d699267a None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:46:46.308657 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c06ed51-0ea6-4321-be85-3a38d699267a None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:46:46.316780 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 621/1250] 10.222.0.27 () {56 vars in 1099 bytes} [Tue Aug 30 01:46:46 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:46:46.322526 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1ca91b3-5e42-4a7c-a123-3fa44e5a2ec4 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:46:46.406232 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:46:46.428346 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1ca91b3-5e42-4a7c-a123-3fa44e5a2ec4 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:46:46.461988 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d36fa4e-35ad-4c86-9fa3-9ed572935aa5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:46:46.462598 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d36fa4e-35ad-4c86-9fa3-9ed572935aa5 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:46:46.462980 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d36fa4e-35ad-4c86-9fa3-9ed572935aa5 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:46:46.463255 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d36fa4e-35ad-4c86-9fa3-9ed572935aa5 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:46:46.472973 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d36fa4e-35ad-4c86-9fa3-9ed572935aa5 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:46:46.474468 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d36fa4e-35ad-4c86-9fa3-9ed572935aa5 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:46:46.489238 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application [None req-6d36fa4e-35ad-4c86-9fa3-9ed572935aa5 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 01:46:46.489238 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 01:46:46.489238 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 01:46:46.489238 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 01:46:46.489238 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 01:46:46.489238 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 01:46:46.489238 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 01:46:46.489238 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 01:46:46.489238 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 01:46:46.489238 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 01:46:46.489238 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 01:46:46.489238 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 01:46:46.489238 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 30 01:46:46.489238 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 30 01:46:46.489238 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 30 01:46:46.489238 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 30 01:46:46.490391 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 01:46:46.490391 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 01:46:46.490391 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 01:46:46.490391 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 01:46:46.490391 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 01:46:46.490391 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 01:46:46.490391 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 01:46:46.490391 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application with Lock( Aug 30 01:46:46.490391 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 01:46:46.490391 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._enter() Aug 30 01:46:46.490391 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 01:46:46.490391 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 01:46:46.490391 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 01:46:46.490391 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.creator() Aug 30 01:46:46.490391 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 01:46:46.490391 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application created_value = creator( Aug 30 01:46:46.490391 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 30 01:46:46.491581 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 30 01:46:46.491581 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 01:46:46.491581 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Aug 30 01:46:46.491581 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 630/1251] 10.222.0.27 () {56 vars in 1108 bytes} [Tue Aug 30 01:46:46 2022] GET /identity/v3/domains/Default => generated 87 bytes in 168 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 01:46:46.496489 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c06ed51-0ea6-4321-be85-3a38d699267a None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:46:46.596546 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c06ed51-0ea6-4321-be85-3a38d699267a None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:46:46.631706 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7b7d574-dd04-4544-a7ad-ec3fb516932d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:46:46.632309 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7b7d574-dd04-4544-a7ad-ec3fb516932d None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:46:46.632582 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7b7d574-dd04-4544-a7ad-ec3fb516932d None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:46:46.632845 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7b7d574-dd04-4544-a7ad-ec3fb516932d None admin] PATH_INFO: `/v3/domains` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:46:46.633495 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7b7d574-dd04-4544-a7ad-ec3fb516932d None admin] RBAC: Adding query filter params (name=Default) {{(pid=81578) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:46:46.633796 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7b7d574-dd04-4544-a7ad-ec3fb516932d None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:46:46.635267 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7b7d574-dd04-4544-a7ad-ec3fb516932d None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:46:46.647041 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 622/1252] 10.222.0.27 () {56 vars in 1109 bytes} [Tue Aug 30 01:46:46 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:46:46.652939 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d36fa4e-35ad-4c86-9fa3-9ed572935aa5 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:46:46.752613 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d36fa4e-35ad-4c86-9fa3-9ed572935aa5 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:46:46.777999 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:46:46.786008 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-986927ff-554d-440d-b981-6451f958f793 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:46:46.786703 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-986927ff-554d-440d-b981-6451f958f793 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:46:46.786979 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-986927ff-554d-440d-b981-6451f958f793 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:46:46.787245 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-986927ff-554d-440d-b981-6451f958f793 None admin] PATH_INFO: `/v3/users/placement` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:46:46.795169 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-986927ff-554d-440d-b981-6451f958f793 None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:46:46.796590 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-986927ff-554d-440d-b981-6451f958f793 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:46:46.808161 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application [None req-986927ff-554d-440d-b981-6451f958f793 None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Aug 30 01:46:46.808161 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 01:46:46.808161 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 01:46:46.808161 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 01:46:46.808161 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 01:46:46.808161 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 01:46:46.808161 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 01:46:46.808161 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 01:46:46.808161 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 01:46:46.808161 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 01:46:46.808161 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 01:46:46.808161 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 01:46:46.808161 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 30 01:46:46.808161 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 30 01:46:46.808161 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 30 01:46:46.808161 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 30 01:46:46.812029 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 01:46:46.812029 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 01:46:46.812029 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 30 01:46:46.812029 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 30 01:46:46.812029 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 30 01:46:46.812029 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 30 01:46:46.812029 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 01:46:46.812029 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 01:46:46.812029 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 01:46:46.812029 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 01:46:46.812029 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 01:46:46.812029 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application with Lock( Aug 30 01:46:46.812029 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 01:46:46.812029 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._enter() Aug 30 01:46:46.812029 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 01:46:46.812029 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 01:46:46.812029 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 01:46:46.812029 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.creator() Aug 30 01:46:46.813032 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 01:46:46.813032 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application created_value = creator( Aug 30 01:46:46.813032 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 30 01:46:46.813032 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 30 01:46:46.813032 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 30 01:46:46.813032 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 30 01:46:46.813032 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 30 01:46:46.813032 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 30 01:46:46.813032 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Aug 30 01:46:46.813032 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Aug 30 01:46:46.813032 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 631/1253] 10.222.0.27 () {56 vars in 1108 bytes} [Tue Aug 30 01:46:46 2022] GET /identity/v3/users/placement => generated 87 bytes in 157 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 01:46:46.815464 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7b7d574-dd04-4544-a7ad-ec3fb516932d None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:46:46.870997 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:46:46.914813 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7b7d574-dd04-4544-a7ad-ec3fb516932d None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:46:46.949202 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-360f97ac-26ef-415d-b011-d0f800c62c00 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:46:46.949867 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-360f97ac-26ef-415d-b011-d0f800c62c00 None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:46:46.950141 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-360f97ac-26ef-415d-b011-d0f800c62c00 None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:46:46.950406 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-360f97ac-26ef-415d-b011-d0f800c62c00 None admin] PATH_INFO: `/v3/users` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:46:46.951123 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-360f97ac-26ef-415d-b011-d0f800c62c00 None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=81578) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:46:46.951420 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-360f97ac-26ef-415d-b011-d0f800c62c00 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:46:46.952820 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-360f97ac-26ef-415d-b011-d0f800c62c00 None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:46:46.970157 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 623/1254] 10.222.0.27 () {56 vars in 1143 bytes} [Tue Aug 30 01:46:46 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:46:46.976567 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-986927ff-554d-440d-b981-6451f958f793 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:46:47.075238 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-986927ff-554d-440d-b981-6451f958f793 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:46:47.107997 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af94cfa1-378a-4615-9b4b-115cb09c9ca8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:46:47.108584 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af94cfa1-378a-4615-9b4b-115cb09c9ca8 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:46:47.108858 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af94cfa1-378a-4615-9b4b-115cb09c9ca8 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:46:47.109126 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af94cfa1-378a-4615-9b4b-115cb09c9ca8 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:46:47.162987 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:46:47.174779 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af94cfa1-378a-4615-9b4b-115cb09c9ca8 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:46:47.177207 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af94cfa1-378a-4615-9b4b-115cb09c9ca8 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:46:47.272441 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 632/1255] 10.222.0.27 () {58 vars in 1282 bytes} [Tue Aug 30 01:46:46 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 296 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Aug 30 01:46:47.278515 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-360f97ac-26ef-415d-b011-d0f800c62c00 None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:46:47.376620 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-360f97ac-26ef-415d-b011-d0f800c62c00 None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:46:47.389502 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:46:47.415813 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce5af352-a29e-4748-9c92-215a2c4ea6db None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:46:47.416372 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce5af352-a29e-4748-9c92-215a2c4ea6db None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:46:47.416650 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce5af352-a29e-4748-9c92-215a2c4ea6db None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:46:47.416919 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce5af352-a29e-4748-9c92-215a2c4ea6db None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:46:47.430330 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce5af352-a29e-4748-9c92-215a2c4ea6db None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:46:47.431769 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce5af352-a29e-4748-9c92-215a2c4ea6db None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:46:47.446021 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application [None req-ce5af352-a29e-4748-9c92-215a2c4ea6db None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 01:46:47.446021 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 01:46:47.446021 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 01:46:47.446021 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 01:46:47.446021 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 01:46:47.446021 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 01:46:47.446021 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 01:46:47.446021 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 01:46:47.446021 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 01:46:47.446021 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 01:46:47.446021 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 01:46:47.446021 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 01:46:47.446021 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 30 01:46:47.446021 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 30 01:46:47.446021 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 30 01:46:47.446021 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 30 01:46:47.447624 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 01:46:47.447624 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 01:46:47.447624 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 01:46:47.447624 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 01:46:47.447624 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 01:46:47.447624 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 01:46:47.447624 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 01:46:47.447624 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application with Lock( Aug 30 01:46:47.447624 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 01:46:47.447624 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._enter() Aug 30 01:46:47.447624 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 01:46:47.447624 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 01:46:47.447624 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 01:46:47.447624 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.creator() Aug 30 01:46:47.447624 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 01:46:47.447624 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application created_value = creator( Aug 30 01:46:47.447624 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 30 01:46:47.449783 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 30 01:46:47.449783 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 01:46:47.449783 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application Aug 30 01:46:47.449783 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 624/1256] 10.222.0.27 () {56 vars in 1108 bytes} [Tue Aug 30 01:46:47 2022] GET /identity/v3/domains/Default => generated 87 bytes in 169 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 01:46:47.454413 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af94cfa1-378a-4615-9b4b-115cb09c9ca8 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:46:47.542762 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:46:47.554979 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af94cfa1-378a-4615-9b4b-115cb09c9ca8 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:46:47.588615 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9611a17-d4e8-4bd0-9e31-1dae5ba09112 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:46:47.589222 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9611a17-d4e8-4bd0-9e31-1dae5ba09112 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:46:47.589526 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9611a17-d4e8-4bd0-9e31-1dae5ba09112 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:46:47.589799 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9611a17-d4e8-4bd0-9e31-1dae5ba09112 None admin] PATH_INFO: `/v3/domains` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:46:47.590447 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9611a17-d4e8-4bd0-9e31-1dae5ba09112 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:46:47.590752 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9611a17-d4e8-4bd0-9e31-1dae5ba09112 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:46:47.592111 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9611a17-d4e8-4bd0-9e31-1dae5ba09112 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:46:47.604066 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 633/1257] 10.222.0.27 () {56 vars in 1109 bytes} [Tue Aug 30 01:46:47 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 30 01:46:47.609880 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce5af352-a29e-4748-9c92-215a2c4ea6db None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:46:47.711548 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce5af352-a29e-4748-9c92-215a2c4ea6db None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:46:47.747985 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfaea2a5-0dcc-421d-974a-34d2d3715332 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:46:47.748428 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfaea2a5-0dcc-421d-974a-34d2d3715332 None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:46:47.748704 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfaea2a5-0dcc-421d-974a-34d2d3715332 None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:46:47.748972 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfaea2a5-0dcc-421d-974a-34d2d3715332 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:46:47.756074 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dfaea2a5-0dcc-421d-974a-34d2d3715332 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:46:47.757540 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dfaea2a5-0dcc-421d-974a-34d2d3715332 None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:46:47.768470 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application [None req-dfaea2a5-0dcc-421d-974a-34d2d3715332 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 30 01:46:47.768470 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 01:46:47.768470 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 01:46:47.768470 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 01:46:47.768470 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 01:46:47.768470 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 01:46:47.768470 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 01:46:47.768470 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 01:46:47.768470 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 01:46:47.768470 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 01:46:47.768470 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 01:46:47.768470 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 01:46:47.768470 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 30 01:46:47.768470 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 30 01:46:47.768470 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 30 01:46:47.773941 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 30 01:46:47.773941 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 01:46:47.773941 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 01:46:47.773941 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 01:46:47.773941 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 01:46:47.773941 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 01:46:47.773941 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 01:46:47.773941 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 01:46:47.773941 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application with Lock( Aug 30 01:46:47.773941 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 01:46:47.773941 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._enter() Aug 30 01:46:47.773941 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 01:46:47.773941 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 01:46:47.773941 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 01:46:47.773941 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.creator() Aug 30 01:46:47.773941 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 01:46:47.773941 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application created_value = creator( Aug 30 01:46:47.775084 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 30 01:46:47.775084 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 30 01:46:47.775084 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 30 01:46:47.775084 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 30 01:46:47.775084 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 30 01:46:47.775084 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 30 01:46:47.775084 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 30 01:46:47.775084 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application Aug 30 01:46:47.775084 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 625/1258] 10.222.0.27 () {56 vars in 1111 bytes} [Tue Aug 30 01:46:47 2022] GET /identity/v3/projects/service => generated 88 bytes in 161 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 01:46:47.776047 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9611a17-d4e8-4bd0-9e31-1dae5ba09112 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:46:47.876153 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9611a17-d4e8-4bd0-9e31-1dae5ba09112 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:46:47.908880 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb3a94c2-fcae-4437-a5d6-55362998054c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:46:47.909727 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb3a94c2-fcae-4437-a5d6-55362998054c None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:46:47.909856 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb3a94c2-fcae-4437-a5d6-55362998054c None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:46:47.910568 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb3a94c2-fcae-4437-a5d6-55362998054c None admin] PATH_INFO: `/v3/projects` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:46:47.911289 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb3a94c2-fcae-4437-a5d6-55362998054c None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:46:47.911593 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb3a94c2-fcae-4437-a5d6-55362998054c None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:46:47.913028 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb3a94c2-fcae-4437-a5d6-55362998054c None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:46:47.924982 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 634/1259] 10.222.0.27 () {56 vars in 1148 bytes} [Tue Aug 30 01:46:47 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 30 01:46:47.931629 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfaea2a5-0dcc-421d-974a-34d2d3715332 None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:46:47.957445 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:46:48.032975 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfaea2a5-0dcc-421d-974a-34d2d3715332 None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:46:48.084329 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c5eb12f-97ce-49f1-a88e-f39a75021bd7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:46:48.084936 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c5eb12f-97ce-49f1-a88e-f39a75021bd7 None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:46:48.085208 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c5eb12f-97ce-49f1-a88e-f39a75021bd7 None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:46:48.085485 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c5eb12f-97ce-49f1-a88e-f39a75021bd7 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:46:48.086200 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c5eb12f-97ce-49f1-a88e-f39a75021bd7 None admin] RBAC: Adding query filter params (role.id=ead5fac86a0747ea8823a82efcea8ed7, scope.project.id=4562f3951fbe465280073e60a11e44e2, user.id=379536ff06fd4f719899e5dde732e63d) {{(pid=81578) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:46:48.086501 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c5eb12f-97ce-49f1-a88e-f39a75021bd7 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:46:48.088017 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c5eb12f-97ce-49f1-a88e-f39a75021bd7 None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:46:48.093507 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 626/1260] 10.222.0.27 () {56 vars in 1374 bytes} [Tue Aug 30 01:46:47 2022] GET /identity/v3/role_assignments?user.id=379536ff06fd4f719899e5dde732e63d&scope.project.id=4562f3951fbe465280073e60a11e44e2&role.id=ead5fac86a0747ea8823a82efcea8ed7 => generated 259 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:46:50.011049 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4d6d49a-4e1e-4eec-b0f4-5ab50e925b53 None None] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:46:50.011560 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4d6d49a-4e1e-4eec-b0f4-5ab50e925b53 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:46:50.011560 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4d6d49a-4e1e-4eec-b0f4-5ab50e925b53 None None] PATH_INFO: `/` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:46:50.012893 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 635/1261] 10.222.0.27 () {54 vars in 927 bytes} [Tue Aug 30 01:46:50 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 01:46:50.018083 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8069804a-6b19-4bae-ac97-34845da45d61 None None] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:46:50.018518 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8069804a-6b19-4bae-ac97-34845da45d61 None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:46:50.018655 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8069804a-6b19-4bae-ac97-34845da45d61 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:46:50.038140 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.auth.core [None req-8069804a-6b19-4bae-ac97-34845da45d61 None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81578) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 01:46:50.039461 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-8069804a-6b19-4bae-ac97-34845da45d61 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:46:50.108865 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 627/1262] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Aug 30 01:46:50 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 30 01:46:50.381596 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4d6d49a-4e1e-4eec-b0f4-5ab50e925b53 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:46:50.392409 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:46:50.484146 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4d6d49a-4e1e-4eec-b0f4-5ab50e925b53 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:46:50.517994 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d6c3ba5-ce5e-4ce4-a279-26ad00227820 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:46:50.518618 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d6c3ba5-ce5e-4ce4-a279-26ad00227820 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:46:50.518896 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d6c3ba5-ce5e-4ce4-a279-26ad00227820 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:46:50.519168 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d6c3ba5-ce5e-4ce4-a279-26ad00227820 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:46:50.529831 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d6c3ba5-ce5e-4ce4-a279-26ad00227820 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 30 01:46:50.534542 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application [None req-7d6c3ba5-ce5e-4ce4-a279-26ad00227820 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 30 01:46:50.534542 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 01:46:50.534542 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 01:46:50.534542 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 01:46:50.534542 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 01:46:50.534542 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 01:46:50.534542 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 01:46:50.534542 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 01:46:50.534542 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 01:46:50.534542 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 01:46:50.534542 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 01:46:50.534542 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 01:46:50.534542 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 30 01:46:50.534542 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 30 01:46:50.534542 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 30 01:46:50.534542 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 30 01:46:50.535662 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 30 01:46:50.535662 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 30 01:46:50.535662 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 30 01:46:50.535662 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 30 01:46:50.535662 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 01:46:50.535662 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 01:46:50.535662 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 01:46:50.535662 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 01:46:50.535662 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 01:46:50.535662 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 01:46:50.535662 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 01:46:50.535662 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application with Lock( Aug 30 01:46:50.535662 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 01:46:50.535662 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._enter() Aug 30 01:46:50.535662 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 01:46:50.535662 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 01:46:50.537442 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 01:46:50.537442 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.creator() Aug 30 01:46:50.537442 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 01:46:50.537442 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application created_value = creator( Aug 30 01:46:50.537442 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 30 01:46:50.537442 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 30 01:46:50.537442 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 30 01:46:50.537442 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 30 01:46:50.537442 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 30 01:46:50.537442 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 30 01:46:50.537442 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 30 01:46:50.537442 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Aug 30 01:46:50.537442 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 636/1263] 10.222.0.27 () {56 vars in 1096 bytes} [Tue Aug 30 01:46:50 2022] GET /identity/v3/roles/admin => generated 83 bytes in 155 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 01:46:50.542466 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8069804a-6b19-4bae-ac97-34845da45d61 None None] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:46:50.560510 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:46:50.639446 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8069804a-6b19-4bae-ac97-34845da45d61 None None] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:46:50.672668 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d30a9f2-e7ac-46f1-8b3b-e41702393e15 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:46:50.673299 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d30a9f2-e7ac-46f1-8b3b-e41702393e15 None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:46:50.673615 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d30a9f2-e7ac-46f1-8b3b-e41702393e15 None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:46:50.673889 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d30a9f2-e7ac-46f1-8b3b-e41702393e15 None admin] PATH_INFO: `/v3/roles` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:46:50.674552 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d30a9f2-e7ac-46f1-8b3b-e41702393e15 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81578) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:46:50.674854 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d30a9f2-e7ac-46f1-8b3b-e41702393e15 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:46:50.676264 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d30a9f2-e7ac-46f1-8b3b-e41702393e15 None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:46:50.684819 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 628/1264] 10.222.0.27 () {56 vars in 1099 bytes} [Tue Aug 30 01:46:50 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 30 01:46:50.690600 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d6c3ba5-ce5e-4ce4-a279-26ad00227820 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:46:50.760214 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:46:50.792914 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d6c3ba5-ce5e-4ce4-a279-26ad00227820 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:46:50.828506 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2b483e5-462a-4da3-a049-195010977055 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:46:50.829123 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2b483e5-462a-4da3-a049-195010977055 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:46:50.829405 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2b483e5-462a-4da3-a049-195010977055 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:46:50.829702 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2b483e5-462a-4da3-a049-195010977055 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:46:50.841190 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2b483e5-462a-4da3-a049-195010977055 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:46:50.842690 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2b483e5-462a-4da3-a049-195010977055 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:46:50.857175 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application [None req-d2b483e5-462a-4da3-a049-195010977055 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 01:46:50.857175 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 01:46:50.857175 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 01:46:50.857175 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 01:46:50.857175 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 01:46:50.857175 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 01:46:50.857175 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 01:46:50.857175 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 01:46:50.857175 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 01:46:50.857175 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 01:46:50.857175 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 01:46:50.857175 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 01:46:50.857175 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 30 01:46:50.857175 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 30 01:46:50.857175 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 30 01:46:50.857175 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 30 01:46:50.858315 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 01:46:50.858315 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 01:46:50.858315 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 01:46:50.858315 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 01:46:50.858315 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 01:46:50.858315 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 01:46:50.858315 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 01:46:50.858315 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application with Lock( Aug 30 01:46:50.858315 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 01:46:50.858315 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._enter() Aug 30 01:46:50.858315 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 01:46:50.858315 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 01:46:50.858315 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 01:46:50.858315 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.creator() Aug 30 01:46:50.858315 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 01:46:50.858315 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application created_value = creator( Aug 30 01:46:50.858315 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 30 01:46:50.859577 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 30 01:46:50.859577 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 01:46:50.859577 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Aug 30 01:46:50.859577 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 637/1265] 10.222.0.27 () {56 vars in 1108 bytes} [Tue Aug 30 01:46:50 2022] GET /identity/v3/domains/Default => generated 87 bytes in 168 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 01:46:50.864099 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d30a9f2-e7ac-46f1-8b3b-e41702393e15 None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:46:50.944368 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:46:50.964100 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d30a9f2-e7ac-46f1-8b3b-e41702393e15 None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:46:50.999129 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03a7a0c1-9c95-4e70-b202-b2f233fbd364 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:46:50.999725 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03a7a0c1-9c95-4e70-b202-b2f233fbd364 None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:46:51.000022 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03a7a0c1-9c95-4e70-b202-b2f233fbd364 None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:46:51.000359 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03a7a0c1-9c95-4e70-b202-b2f233fbd364 None admin] PATH_INFO: `/v3/domains` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:46:51.000930 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03a7a0c1-9c95-4e70-b202-b2f233fbd364 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81578) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:46:51.001229 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03a7a0c1-9c95-4e70-b202-b2f233fbd364 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:46:51.002609 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03a7a0c1-9c95-4e70-b202-b2f233fbd364 None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:46:51.013788 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 629/1266] 10.222.0.27 () {56 vars in 1109 bytes} [Tue Aug 30 01:46:50 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:46:51.019677 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2b483e5-462a-4da3-a049-195010977055 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:46:51.121035 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2b483e5-462a-4da3-a049-195010977055 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:46:51.141134 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:46:51.155980 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69be3739-c9f8-417c-8ca2-d032086ed19e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:46:51.156594 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69be3739-c9f8-417c-8ca2-d032086ed19e None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:46:51.156872 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69be3739-c9f8-417c-8ca2-d032086ed19e None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:46:51.157144 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69be3739-c9f8-417c-8ca2-d032086ed19e None admin] PATH_INFO: `/v3/users/placement` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:46:51.165504 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69be3739-c9f8-417c-8ca2-d032086ed19e None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:46:51.166950 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69be3739-c9f8-417c-8ca2-d032086ed19e None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:46:51.177913 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application [None req-69be3739-c9f8-417c-8ca2-d032086ed19e None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Aug 30 01:46:51.177913 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 01:46:51.177913 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 01:46:51.177913 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 01:46:51.177913 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 01:46:51.177913 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 01:46:51.177913 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 01:46:51.177913 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 01:46:51.177913 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 01:46:51.177913 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 01:46:51.177913 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 01:46:51.177913 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 01:46:51.177913 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 30 01:46:51.177913 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 30 01:46:51.177913 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 30 01:46:51.177913 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 30 01:46:51.179036 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 01:46:51.179036 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 01:46:51.179036 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 30 01:46:51.179036 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 30 01:46:51.179036 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 30 01:46:51.179036 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 30 01:46:51.179036 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 01:46:51.179036 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 01:46:51.179036 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 01:46:51.179036 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 01:46:51.179036 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 01:46:51.179036 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application with Lock( Aug 30 01:46:51.179036 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 01:46:51.179036 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._enter() Aug 30 01:46:51.179036 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 01:46:51.179036 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 01:46:51.179036 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 01:46:51.179036 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.creator() Aug 30 01:46:51.180207 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 01:46:51.180207 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application created_value = creator( Aug 30 01:46:51.180207 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 30 01:46:51.180207 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 30 01:46:51.180207 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 30 01:46:51.180207 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 30 01:46:51.180207 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 30 01:46:51.180207 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 30 01:46:51.180207 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Aug 30 01:46:51.180207 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Aug 30 01:46:51.180207 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 638/1267] 10.222.0.27 () {56 vars in 1108 bytes} [Tue Aug 30 01:46:51 2022] GET /identity/v3/users/placement => generated 87 bytes in 160 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 01:46:51.185009 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03a7a0c1-9c95-4e70-b202-b2f233fbd364 None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:46:51.283447 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03a7a0c1-9c95-4e70-b202-b2f233fbd364 None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:46:51.318549 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1180e9b1-123b-449c-9856-c61b90c3d19b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:46:51.319180 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1180e9b1-123b-449c-9856-c61b90c3d19b None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:46:51.319460 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1180e9b1-123b-449c-9856-c61b90c3d19b None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:46:51.319728 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1180e9b1-123b-449c-9856-c61b90c3d19b None admin] PATH_INFO: `/v3/users` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:46:51.320452 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1180e9b1-123b-449c-9856-c61b90c3d19b None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=81578) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:46:51.320752 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1180e9b1-123b-449c-9856-c61b90c3d19b None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:46:51.322196 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1180e9b1-123b-449c-9856-c61b90c3d19b None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:46:51.330846 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:46:51.339849 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 630/1268] 10.222.0.27 () {56 vars in 1143 bytes} [Tue Aug 30 01:46:51 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:46:51.345654 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69be3739-c9f8-417c-8ca2-d032086ed19e None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:46:51.454814 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69be3739-c9f8-417c-8ca2-d032086ed19e None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:46:51.494830 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0af154b5-312b-4f07-9359-84f674fb0af6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:46:51.495457 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0af154b5-312b-4f07-9359-84f674fb0af6 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:46:51.495739 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0af154b5-312b-4f07-9359-84f674fb0af6 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:46:51.496012 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0af154b5-312b-4f07-9359-84f674fb0af6 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:46:51.506885 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0af154b5-312b-4f07-9359-84f674fb0af6 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:46:51.508348 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0af154b5-312b-4f07-9359-84f674fb0af6 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:46:51.523181 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application [None req-0af154b5-312b-4f07-9359-84f674fb0af6 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 01:46:51.523181 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 01:46:51.523181 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 01:46:51.523181 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 01:46:51.523181 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 01:46:51.523181 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 01:46:51.523181 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 01:46:51.523181 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 01:46:51.523181 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 01:46:51.523181 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 01:46:51.523181 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 01:46:51.523181 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 01:46:51.523181 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 30 01:46:51.523181 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 30 01:46:51.523181 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 30 01:46:51.523181 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 30 01:46:51.524359 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 01:46:51.524359 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 01:46:51.524359 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 01:46:51.524359 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 01:46:51.524359 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 01:46:51.524359 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 01:46:51.524359 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 01:46:51.524359 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application with Lock( Aug 30 01:46:51.524359 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 01:46:51.524359 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._enter() Aug 30 01:46:51.524359 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 01:46:51.524359 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 01:46:51.524359 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 01:46:51.524359 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.creator() Aug 30 01:46:51.524359 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 01:46:51.524359 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application created_value = creator( Aug 30 01:46:51.524359 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 30 01:46:51.525843 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 30 01:46:51.525843 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 01:46:51.525843 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Aug 30 01:46:51.525843 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 639/1269] 10.222.0.27 () {56 vars in 1108 bytes} [Tue Aug 30 01:46:51 2022] GET /identity/v3/domains/Default => generated 87 bytes in 179 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 01:46:51.530136 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1180e9b1-123b-449c-9856-c61b90c3d19b None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:46:51.629350 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1180e9b1-123b-449c-9856-c61b90c3d19b None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:46:51.663227 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0006cb7-0714-41f5-9eac-dfae77f3291a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:46:51.663830 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0006cb7-0714-41f5-9eac-dfae77f3291a None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:46:51.664104 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0006cb7-0714-41f5-9eac-dfae77f3291a None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:46:51.664371 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0006cb7-0714-41f5-9eac-dfae77f3291a None admin] PATH_INFO: `/v3/domains` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:46:51.665008 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0006cb7-0714-41f5-9eac-dfae77f3291a None admin] RBAC: Adding query filter params (name=Default) {{(pid=81578) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:46:51.665308 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0006cb7-0714-41f5-9eac-dfae77f3291a None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:46:51.666691 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0006cb7-0714-41f5-9eac-dfae77f3291a None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:46:51.678548 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 631/1270] 10.222.0.27 () {56 vars in 1109 bytes} [Tue Aug 30 01:46:51 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:46:51.684413 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0af154b5-312b-4f07-9359-84f674fb0af6 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:46:51.719972 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:46:51.789601 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0af154b5-312b-4f07-9359-84f674fb0af6 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:46:51.823178 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5f45603-3c49-45f5-a44c-b6f8c16ac7ec None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:46:51.823807 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5f45603-3c49-45f5-a44c-b6f8c16ac7ec None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:46:51.824087 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5f45603-3c49-45f5-a44c-b6f8c16ac7ec None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:46:51.824357 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5f45603-3c49-45f5-a44c-b6f8c16ac7ec None admin] PATH_INFO: `/v3/projects/service` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:46:51.830861 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5f45603-3c49-45f5-a44c-b6f8c16ac7ec None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:46:51.832297 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5f45603-3c49-45f5-a44c-b6f8c16ac7ec None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:46:51.845013 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application [None req-d5f45603-3c49-45f5-a44c-b6f8c16ac7ec None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 30 01:46:51.845013 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 01:46:51.845013 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 01:46:51.845013 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 01:46:51.845013 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 01:46:51.845013 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 01:46:51.845013 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 01:46:51.845013 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 01:46:51.845013 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 01:46:51.845013 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 01:46:51.845013 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 01:46:51.845013 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 01:46:51.845013 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 30 01:46:51.845013 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 30 01:46:51.845013 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 30 01:46:51.850605 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 30 01:46:51.850605 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 01:46:51.850605 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 01:46:51.850605 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 01:46:51.850605 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 01:46:51.850605 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 01:46:51.850605 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 01:46:51.850605 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 01:46:51.850605 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application with Lock( Aug 30 01:46:51.850605 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 01:46:51.850605 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._enter() Aug 30 01:46:51.850605 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 01:46:51.850605 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 01:46:51.850605 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 01:46:51.850605 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.creator() Aug 30 01:46:51.850605 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 01:46:51.850605 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application created_value = creator( Aug 30 01:46:51.851725 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 30 01:46:51.851725 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 30 01:46:51.851725 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 30 01:46:51.851725 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 30 01:46:51.851725 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 30 01:46:51.851725 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 30 01:46:51.851725 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 30 01:46:51.851725 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Aug 30 01:46:51.851725 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 640/1271] 10.222.0.27 () {56 vars in 1111 bytes} [Tue Aug 30 01:46:51 2022] GET /identity/v3/projects/service => generated 88 bytes in 162 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 01:46:51.852823 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0006cb7-0714-41f5-9eac-dfae77f3291a None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:46:51.910241 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:46:51.950631 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0006cb7-0714-41f5-9eac-dfae77f3291a None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:46:51.983478 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a59aa647-87e5-4d1b-8301-d06be5842ee3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:46:51.984073 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a59aa647-87e5-4d1b-8301-d06be5842ee3 None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:46:51.984349 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a59aa647-87e5-4d1b-8301-d06be5842ee3 None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:46:51.984618 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a59aa647-87e5-4d1b-8301-d06be5842ee3 None admin] PATH_INFO: `/v3/projects` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:46:51.985302 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a59aa647-87e5-4d1b-8301-d06be5842ee3 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81578) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:46:51.985635 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a59aa647-87e5-4d1b-8301-d06be5842ee3 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:46:51.987039 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a59aa647-87e5-4d1b-8301-d06be5842ee3 None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:46:51.998489 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 632/1272] 10.222.0.27 () {56 vars in 1148 bytes} [Tue Aug 30 01:46:51 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:46:52.005230 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5f45603-3c49-45f5-a44c-b6f8c16ac7ec None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:46:52.104174 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:46:52.112464 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5f45603-3c49-45f5-a44c-b6f8c16ac7ec None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:46:52.146558 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6da81896-fa5a-4de6-9027-65edc28c7c55 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:46:52.147225 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6da81896-fa5a-4de6-9027-65edc28c7c55 None admin] REQUEST_METHOD: `PUT` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:46:52.147502 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6da81896-fa5a-4de6-9027-65edc28c7c55 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:46:52.147770 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6da81896-fa5a-4de6-9027-65edc28c7c55 None admin] PATH_INFO: `/v3/projects/4562f3951fbe465280073e60a11e44e2/users/379536ff06fd4f719899e5dde732e63d/roles/ead5fac86a0747ea8823a82efcea8ed7` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:46:52.149747 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6da81896-fa5a-4de6-9027-65edc28c7c55 None admin] RBAC: Authorizing `identity:create_grant(project_id=4562f3951fbe465280073e60a11e44e2, user_id=379536ff06fd4f719899e5dde732e63d, role_id=ead5fac86a0747ea8823a82efcea8ed7)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:46:52.154812 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6da81896-fa5a-4de6-9027-65edc28c7c55 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:46:52.251078 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 641/1273] 10.222.0.27 () {58 vars in 1439 bytes} [Tue Aug 30 01:46:52 2022] PUT /identity/v3/projects/4562f3951fbe465280073e60a11e44e2/users/379536ff06fd4f719899e5dde732e63d/roles/ead5fac86a0747ea8823a82efcea8ed7 => generated 0 bytes in 246 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 01:46:54.131981 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-666c8257-cc16-439d-a3ef-7a371ec0d357 None None] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:46:54.132517 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-666c8257-cc16-439d-a3ef-7a371ec0d357 None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:46:54.132517 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-666c8257-cc16-439d-a3ef-7a371ec0d357 None None] PATH_INFO: `/` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:46:54.133878 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 633/1274] 10.222.0.27 () {54 vars in 927 bytes} [Tue Aug 30 01:46:54 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 01:46:54.138957 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd43e6b1-06c7-4c00-baff-90f37bb3d383 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:46:54.139354 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd43e6b1-06c7-4c00-baff-90f37bb3d383 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:46:54.139503 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd43e6b1-06c7-4c00-baff-90f37bb3d383 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:46:54.157952 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-dd43e6b1-06c7-4c00-baff-90f37bb3d383 None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 01:46:54.159136 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-dd43e6b1-06c7-4c00-baff-90f37bb3d383 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:46:54.226077 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 642/1275] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Aug 30 01:46:54 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 30 01:46:54.494763 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-666c8257-cc16-439d-a3ef-7a371ec0d357 None None] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:46:54.595703 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-666c8257-cc16-439d-a3ef-7a371ec0d357 None None] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:46:54.614012 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:46:54.630500 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-380fc7c3-5126-46df-9b9b-58cf885cba98 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:46:54.631118 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-380fc7c3-5126-46df-9b9b-58cf885cba98 None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:46:54.631397 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-380fc7c3-5126-46df-9b9b-58cf885cba98 None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:46:54.631666 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-380fc7c3-5126-46df-9b9b-58cf885cba98 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:46:54.641876 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-380fc7c3-5126-46df-9b9b-58cf885cba98 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 30 01:46:54.647298 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application [None req-380fc7c3-5126-46df-9b9b-58cf885cba98 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 30 01:46:54.647298 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 01:46:54.647298 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 01:46:54.647298 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 01:46:54.647298 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 01:46:54.647298 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 01:46:54.647298 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 01:46:54.647298 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 01:46:54.647298 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 01:46:54.647298 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 01:46:54.647298 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 01:46:54.647298 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 01:46:54.647298 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 30 01:46:54.647298 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 30 01:46:54.647298 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 30 01:46:54.647298 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 30 01:46:54.648494 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 30 01:46:54.648494 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 30 01:46:54.648494 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 30 01:46:54.648494 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 30 01:46:54.648494 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 01:46:54.648494 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 01:46:54.648494 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 01:46:54.648494 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 01:46:54.648494 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 01:46:54.648494 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 01:46:54.648494 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 01:46:54.648494 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application with Lock( Aug 30 01:46:54.648494 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 01:46:54.648494 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._enter() Aug 30 01:46:54.648494 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 01:46:54.648494 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 01:46:54.652264 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 01:46:54.652264 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.creator() Aug 30 01:46:54.652264 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 01:46:54.652264 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application created_value = creator( Aug 30 01:46:54.652264 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 30 01:46:54.652264 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 30 01:46:54.652264 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 30 01:46:54.652264 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 30 01:46:54.652264 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 30 01:46:54.652264 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 30 01:46:54.652264 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 30 01:46:54.652264 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application Aug 30 01:46:54.652264 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 634/1276] 10.222.0.27 () {56 vars in 1096 bytes} [Tue Aug 30 01:46:54 2022] GET /identity/v3/roles/admin => generated 83 bytes in 154 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 01:46:54.654699 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd43e6b1-06c7-4c00-baff-90f37bb3d383 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:46:54.738548 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:46:54.756721 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd43e6b1-06c7-4c00-baff-90f37bb3d383 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:46:54.789750 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77f9c577-0fab-4c30-9707-3c774d3fa8d1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:46:54.790346 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77f9c577-0fab-4c30-9707-3c774d3fa8d1 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:46:54.790627 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77f9c577-0fab-4c30-9707-3c774d3fa8d1 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:46:54.790893 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77f9c577-0fab-4c30-9707-3c774d3fa8d1 None admin] PATH_INFO: `/v3/roles` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:46:54.791548 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77f9c577-0fab-4c30-9707-3c774d3fa8d1 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:46:54.791845 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77f9c577-0fab-4c30-9707-3c774d3fa8d1 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:46:54.793190 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77f9c577-0fab-4c30-9707-3c774d3fa8d1 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:46:54.801045 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 643/1277] 10.222.0.27 () {56 vars in 1099 bytes} [Tue Aug 30 01:46:54 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:46:54.807056 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-380fc7c3-5126-46df-9b9b-58cf885cba98 None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:46:54.905615 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-380fc7c3-5126-46df-9b9b-58cf885cba98 None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:46:54.938256 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f086cacd-4ad9-4e46-adb2-7f2f07f99a2a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:46:54.938870 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f086cacd-4ad9-4e46-adb2-7f2f07f99a2a None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:46:54.939144 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f086cacd-4ad9-4e46-adb2-7f2f07f99a2a None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:46:54.939409 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f086cacd-4ad9-4e46-adb2-7f2f07f99a2a None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:46:54.949657 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f086cacd-4ad9-4e46-adb2-7f2f07f99a2a None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:46:54.951076 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f086cacd-4ad9-4e46-adb2-7f2f07f99a2a None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:46:54.966596 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application [None req-f086cacd-4ad9-4e46-adb2-7f2f07f99a2a None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 01:46:54.966596 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 01:46:54.966596 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 01:46:54.966596 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 01:46:54.966596 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 01:46:54.966596 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 01:46:54.966596 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 01:46:54.966596 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 01:46:54.966596 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 01:46:54.966596 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 01:46:54.966596 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 01:46:54.966596 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 01:46:54.966596 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 30 01:46:54.966596 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 30 01:46:54.966596 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 30 01:46:54.966596 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 30 01:46:54.967730 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 01:46:54.967730 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 01:46:54.967730 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 01:46:54.967730 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 01:46:54.967730 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 01:46:54.967730 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 01:46:54.967730 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 01:46:54.967730 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application with Lock( Aug 30 01:46:54.967730 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 01:46:54.967730 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._enter() Aug 30 01:46:54.967730 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 01:46:54.967730 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 01:46:54.967730 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 01:46:54.967730 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.creator() Aug 30 01:46:54.967730 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 01:46:54.967730 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application created_value = creator( Aug 30 01:46:54.967730 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 30 01:46:54.968833 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 30 01:46:54.968833 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 01:46:54.968833 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application Aug 30 01:46:54.968833 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 635/1278] 10.222.0.27 () {56 vars in 1108 bytes} [Tue Aug 30 01:46:54 2022] GET /identity/v3/domains/Default => generated 87 bytes in 161 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 01:46:54.973916 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77f9c577-0fab-4c30-9707-3c774d3fa8d1 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:46:55.074258 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77f9c577-0fab-4c30-9707-3c774d3fa8d1 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:46:55.107639 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98174ff5-b3e0-4105-8864-9f303cb96d53 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:46:55.108269 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98174ff5-b3e0-4105-8864-9f303cb96d53 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:46:55.108547 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98174ff5-b3e0-4105-8864-9f303cb96d53 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:46:55.108847 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98174ff5-b3e0-4105-8864-9f303cb96d53 None admin] PATH_INFO: `/v3/domains` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:46:55.109506 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98174ff5-b3e0-4105-8864-9f303cb96d53 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:46:55.109809 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98174ff5-b3e0-4105-8864-9f303cb96d53 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:46:55.111177 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98174ff5-b3e0-4105-8864-9f303cb96d53 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:46:55.123239 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 644/1279] 10.222.0.27 () {56 vars in 1109 bytes} [Tue Aug 30 01:46:54 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:46:55.128963 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f086cacd-4ad9-4e46-adb2-7f2f07f99a2a None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:46:55.159752 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:46:55.227696 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f086cacd-4ad9-4e46-adb2-7f2f07f99a2a None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:46:55.260551 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd842316-a494-4b20-a81b-71ac7e91e983 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:46:55.261237 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd842316-a494-4b20-a81b-71ac7e91e983 None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:46:55.261543 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd842316-a494-4b20-a81b-71ac7e91e983 None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:46:55.261811 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd842316-a494-4b20-a81b-71ac7e91e983 None admin] PATH_INFO: `/v3/users/placement` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:46:55.269004 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd842316-a494-4b20-a81b-71ac7e91e983 None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:46:55.270691 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd842316-a494-4b20-a81b-71ac7e91e983 None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:46:55.281652 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application [None req-cd842316-a494-4b20-a81b-71ac7e91e983 None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Aug 30 01:46:55.281652 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 01:46:55.281652 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 01:46:55.281652 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 01:46:55.281652 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 01:46:55.281652 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 01:46:55.281652 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 01:46:55.281652 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 01:46:55.281652 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 01:46:55.281652 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 01:46:55.281652 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 01:46:55.281652 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 01:46:55.281652 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 30 01:46:55.281652 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 30 01:46:55.281652 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 30 01:46:55.281652 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 30 01:46:55.282668 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 01:46:55.282668 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 01:46:55.282668 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 30 01:46:55.282668 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 30 01:46:55.282668 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 30 01:46:55.282668 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 30 01:46:55.282668 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 01:46:55.282668 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 01:46:55.282668 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 01:46:55.282668 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 01:46:55.282668 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 01:46:55.282668 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application with Lock( Aug 30 01:46:55.282668 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 01:46:55.282668 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._enter() Aug 30 01:46:55.282668 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 01:46:55.282668 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 01:46:55.282668 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 01:46:55.282668 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.creator() Aug 30 01:46:55.283950 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 01:46:55.283950 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application created_value = creator( Aug 30 01:46:55.283950 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 30 01:46:55.283950 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 30 01:46:55.283950 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 30 01:46:55.283950 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 30 01:46:55.283950 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 30 01:46:55.283950 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 30 01:46:55.283950 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Aug 30 01:46:55.283950 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application Aug 30 01:46:55.283950 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 636/1280] 10.222.0.27 () {56 vars in 1108 bytes} [Tue Aug 30 01:46:55 2022] GET /identity/v3/users/placement => generated 87 bytes in 154 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 01:46:55.288992 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98174ff5-b3e0-4105-8864-9f303cb96d53 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:46:55.292968 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:46:55.388787 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98174ff5-b3e0-4105-8864-9f303cb96d53 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:46:55.422565 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87d108a0-8b02-40fa-9031-caea977c97fe None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:46:55.423195 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87d108a0-8b02-40fa-9031-caea977c97fe None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:46:55.423528 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87d108a0-8b02-40fa-9031-caea977c97fe None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:46:55.423797 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87d108a0-8b02-40fa-9031-caea977c97fe None admin] PATH_INFO: `/v3/users` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:46:55.424515 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87d108a0-8b02-40fa-9031-caea977c97fe None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:46:55.424814 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87d108a0-8b02-40fa-9031-caea977c97fe None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:46:55.426830 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87d108a0-8b02-40fa-9031-caea977c97fe None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:46:55.443923 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 645/1281] 10.222.0.27 () {56 vars in 1143 bytes} [Tue Aug 30 01:46:55 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:46:55.450041 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd842316-a494-4b20-a81b-71ac7e91e983 None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:46:55.540327 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:46:55.549615 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd842316-a494-4b20-a81b-71ac7e91e983 None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:46:55.582045 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a67c421d-b340-477d-b08f-c1df10c55bec None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:46:55.582672 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a67c421d-b340-477d-b08f-c1df10c55bec None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:46:55.582953 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a67c421d-b340-477d-b08f-c1df10c55bec None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:46:55.583428 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a67c421d-b340-477d-b08f-c1df10c55bec None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:46:55.649710 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a67c421d-b340-477d-b08f-c1df10c55bec None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:46:55.651080 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a67c421d-b340-477d-b08f-c1df10c55bec None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:46:55.741337 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:46:55.751522 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 637/1282] 10.222.0.27 () {58 vars in 1282 bytes} [Tue Aug 30 01:46:55 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 302 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Aug 30 01:46:55.757539 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87d108a0-8b02-40fa-9031-caea977c97fe None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:46:55.820412 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:46:55.858616 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87d108a0-8b02-40fa-9031-caea977c97fe None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:46:55.891639 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b44058df-6d48-4ad1-a80f-3f51eb6e2a6f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:46:55.892256 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b44058df-6d48-4ad1-a80f-3f51eb6e2a6f None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:46:55.892533 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b44058df-6d48-4ad1-a80f-3f51eb6e2a6f None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:46:55.892803 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b44058df-6d48-4ad1-a80f-3f51eb6e2a6f None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:46:55.902546 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b44058df-6d48-4ad1-a80f-3f51eb6e2a6f None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:46:55.903961 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b44058df-6d48-4ad1-a80f-3f51eb6e2a6f None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:46:55.918663 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application [None req-b44058df-6d48-4ad1-a80f-3f51eb6e2a6f None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 01:46:55.918663 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 01:46:55.918663 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 01:46:55.918663 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 01:46:55.918663 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 01:46:55.918663 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 01:46:55.918663 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 01:46:55.918663 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 01:46:55.918663 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 01:46:55.918663 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 01:46:55.918663 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 01:46:55.918663 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 01:46:55.918663 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 30 01:46:55.918663 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 30 01:46:55.918663 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 30 01:46:55.918663 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 30 01:46:55.919783 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 01:46:55.919783 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 01:46:55.919783 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 01:46:55.919783 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 01:46:55.919783 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 01:46:55.919783 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 01:46:55.919783 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 01:46:55.919783 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application with Lock( Aug 30 01:46:55.919783 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 01:46:55.919783 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._enter() Aug 30 01:46:55.919783 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 01:46:55.919783 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 01:46:55.919783 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 01:46:55.919783 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.creator() Aug 30 01:46:55.919783 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 01:46:55.919783 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application created_value = creator( Aug 30 01:46:55.919783 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 30 01:46:55.920984 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 30 01:46:55.920984 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 30 01:46:55.920984 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Aug 30 01:46:55.921616 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 646/1283] 10.222.0.27 () {56 vars in 1108 bytes} [Tue Aug 30 01:46:55 2022] GET /identity/v3/domains/Default => generated 87 bytes in 165 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 01:46:55.925399 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a67c421d-b340-477d-b08f-c1df10c55bec None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:46:56.027544 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a67c421d-b340-477d-b08f-c1df10c55bec None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:46:56.061390 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26ad5de7-c677-4aa9-82d5-2aaee9b178b3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:46:56.062026 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26ad5de7-c677-4aa9-82d5-2aaee9b178b3 None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:46:56.062305 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26ad5de7-c677-4aa9-82d5-2aaee9b178b3 None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:46:56.062577 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26ad5de7-c677-4aa9-82d5-2aaee9b178b3 None admin] PATH_INFO: `/v3/domains` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:46:56.063214 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26ad5de7-c677-4aa9-82d5-2aaee9b178b3 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81578) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:46:56.063555 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26ad5de7-c677-4aa9-82d5-2aaee9b178b3 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:46:56.064872 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26ad5de7-c677-4aa9-82d5-2aaee9b178b3 None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:46:56.075833 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 638/1284] 10.222.0.27 () {56 vars in 1109 bytes} [Tue Aug 30 01:46:55 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:46:56.081510 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b44058df-6d48-4ad1-a80f-3f51eb6e2a6f None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:46:56.181550 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b44058df-6d48-4ad1-a80f-3f51eb6e2a6f None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:46:56.194853 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:46:56.214544 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3b6fdba-e714-4c01-8bef-dae5223be012 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:46:56.215166 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3b6fdba-e714-4c01-8bef-dae5223be012 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:46:56.215446 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3b6fdba-e714-4c01-8bef-dae5223be012 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:46:56.215718 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3b6fdba-e714-4c01-8bef-dae5223be012 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:46:56.222162 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3b6fdba-e714-4c01-8bef-dae5223be012 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:46:56.224501 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3b6fdba-e714-4c01-8bef-dae5223be012 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:46:56.236109 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application [None req-d3b6fdba-e714-4c01-8bef-dae5223be012 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 30 01:46:56.236109 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 01:46:56.236109 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 01:46:56.236109 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 01:46:56.236109 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 01:46:56.236109 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 01:46:56.236109 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 01:46:56.236109 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 01:46:56.236109 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 01:46:56.236109 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 01:46:56.236109 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 01:46:56.236109 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 01:46:56.236109 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 30 01:46:56.236109 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 30 01:46:56.236109 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 30 01:46:56.237194 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 30 01:46:56.237194 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 01:46:56.237194 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 01:46:56.237194 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 01:46:56.237194 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 01:46:56.237194 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 01:46:56.237194 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 01:46:56.237194 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 01:46:56.237194 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application with Lock( Aug 30 01:46:56.237194 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 01:46:56.237194 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._enter() Aug 30 01:46:56.237194 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 01:46:56.237194 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 01:46:56.237194 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 01:46:56.237194 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.creator() Aug 30 01:46:56.237194 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 01:46:56.237194 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application created_value = creator( Aug 30 01:46:56.238411 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 30 01:46:56.238411 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 30 01:46:56.238411 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 30 01:46:56.238411 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 30 01:46:56.238411 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 30 01:46:56.238411 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 30 01:46:56.238411 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 30 01:46:56.238411 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Aug 30 01:46:56.238411 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 647/1285] 10.222.0.27 () {56 vars in 1111 bytes} [Tue Aug 30 01:46:56 2022] GET /identity/v3/projects/service => generated 88 bytes in 156 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 01:46:56.244817 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26ad5de7-c677-4aa9-82d5-2aaee9b178b3 None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:46:56.295687 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:46:56.359018 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26ad5de7-c677-4aa9-82d5-2aaee9b178b3 None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:46:56.396836 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6978dbfc-08e5-4230-99be-44170d54ac49 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:46:56.397450 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6978dbfc-08e5-4230-99be-44170d54ac49 None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:46:56.397755 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6978dbfc-08e5-4230-99be-44170d54ac49 None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:46:56.398022 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6978dbfc-08e5-4230-99be-44170d54ac49 None admin] PATH_INFO: `/v3/projects` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:46:56.398725 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6978dbfc-08e5-4230-99be-44170d54ac49 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81578) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:46:56.399029 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6978dbfc-08e5-4230-99be-44170d54ac49 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:46:56.400562 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6978dbfc-08e5-4230-99be-44170d54ac49 None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:46:56.414836 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 639/1286] 10.222.0.27 () {56 vars in 1148 bytes} [Tue Aug 30 01:46:56 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:46:56.421147 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3b6fdba-e714-4c01-8bef-dae5223be012 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:46:56.528950 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3b6fdba-e714-4c01-8bef-dae5223be012 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:46:56.565916 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f30bbb07-8e8b-4880-b8a5-dcb159f02d14 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:46:56.566537 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f30bbb07-8e8b-4880-b8a5-dcb159f02d14 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:46:56.566813 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f30bbb07-8e8b-4880-b8a5-dcb159f02d14 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:46:56.567080 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f30bbb07-8e8b-4880-b8a5-dcb159f02d14 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:46:56.567861 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f30bbb07-8e8b-4880-b8a5-dcb159f02d14 None admin] RBAC: Adding query filter params (role.id=ead5fac86a0747ea8823a82efcea8ed7, scope.project.id=4562f3951fbe465280073e60a11e44e2, user.id=379536ff06fd4f719899e5dde732e63d) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:46:56.568223 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f30bbb07-8e8b-4880-b8a5-dcb159f02d14 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:46:56.569697 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f30bbb07-8e8b-4880-b8a5-dcb159f02d14 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:46:56.585491 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 648/1287] 10.222.0.27 () {56 vars in 1374 bytes} [Tue Aug 30 01:46:56 2022] GET /identity/v3/role_assignments?user.id=379536ff06fd4f719899e5dde732e63d&scope.project.id=4562f3951fbe465280073e60a11e44e2&role.id=ead5fac86a0747ea8823a82efcea8ed7 => generated 608 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:46:58.595990 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d010dd9-ec44-43a9-a011-031e32c66823 None None] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:46:58.596556 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d010dd9-ec44-43a9-a011-031e32c66823 None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:46:58.596725 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d010dd9-ec44-43a9-a011-031e32c66823 None None] PATH_INFO: `/` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:46:58.598834 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 640/1288] 10.222.0.27 () {54 vars in 927 bytes} [Tue Aug 30 01:46:58 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 01:46:58.604296 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be25642f-c6c3-4bb8-8ac1-c70a9bd099ea None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:46:58.604704 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be25642f-c6c3-4bb8-8ac1-c70a9bd099ea None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:46:58.604840 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be25642f-c6c3-4bb8-8ac1-c70a9bd099ea None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:46:58.623097 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-be25642f-c6c3-4bb8-8ac1-c70a9bd099ea None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 01:46:58.624371 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-be25642f-c6c3-4bb8-8ac1-c70a9bd099ea None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:46:58.654002 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:46:58.690475 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 649/1289] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Aug 30 01:46:58 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 30 01:46:58.958786 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d010dd9-ec44-43a9-a011-031e32c66823 None None] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:46:59.056582 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d010dd9-ec44-43a9-a011-031e32c66823 None None] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:46:59.067696 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:46:59.090833 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bd3031b-841a-4c30-a2a6-03a572c9ce12 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:46:59.091473 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bd3031b-841a-4c30-a2a6-03a572c9ce12 None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:46:59.091750 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bd3031b-841a-4c30-a2a6-03a572c9ce12 None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:46:59.092015 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bd3031b-841a-4c30-a2a6-03a572c9ce12 None admin] PATH_INFO: `/v3/services/placement` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:46:59.092588 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4bd3031b-841a-4c30-a2a6-03a572c9ce12 None admin] RBAC: Authorizing `identity:get_service(service_id=placement)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:46:59.093964 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4bd3031b-841a-4c30-a2a6-03a572c9ce12 None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:46:59.103226 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application [None req-4bd3031b-841a-4c30-a2a6-03a572c9ce12 None admin] Could not find service: placement.: keystone.exception.ServiceNotFound: Could not find service: placement. Aug 30 01:46:59.103226 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 01:46:59.103226 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Aug 30 01:46:59.103226 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Aug 30 01:46:59.103226 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Aug 30 01:46:59.103226 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Aug 30 01:46:59.103226 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Aug 30 01:46:59.103226 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 30 01:46:59.103226 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Aug 30 01:46:59.103226 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application Aug 30 01:46:59.103226 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 30 01:46:59.103226 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application Aug 30 01:46:59.103226 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 01:46:59.103226 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 01:46:59.103226 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 01:46:59.103226 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 01:46:59.103226 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 01:46:59.104523 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 01:46:59.104523 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 01:46:59.104523 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 01:46:59.104523 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 01:46:59.104523 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 01:46:59.104523 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 01:46:59.104523 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Aug 30 01:46:59.104523 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._get_service(service_id) Aug 30 01:46:59.104523 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Aug 30 01:46:59.104523 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Aug 30 01:46:59.104523 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 01:46:59.104523 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 01:46:59.104523 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 01:46:59.104523 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 01:46:59.104523 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 01:46:59.104523 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 01:46:59.105823 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 01:46:59.105823 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application with Lock( Aug 30 01:46:59.105823 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 01:46:59.105823 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._enter() Aug 30 01:46:59.105823 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 01:46:59.105823 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 01:46:59.105823 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 01:46:59.105823 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.creator() Aug 30 01:46:59.105823 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 01:46:59.105823 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application created_value = creator( Aug 30 01:46:59.105823 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Aug 30 01:46:59.105823 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 30 01:46:59.105823 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Aug 30 01:46:59.105823 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application Aug 30 01:46:59.105823 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 641/1290] 10.222.0.27 () {56 vars in 1117 bytes} [Tue Aug 30 01:46:58 2022] GET /identity/v3/services/placement => generated 90 bytes in 147 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 01:46:59.110346 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be25642f-c6c3-4bb8-8ac1-c70a9bd099ea None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:46:59.209102 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be25642f-c6c3-4bb8-8ac1-c70a9bd099ea None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:46:59.243473 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e6267b4-e0af-41df-8218-617670a8f3de None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:46:59.244072 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e6267b4-e0af-41df-8218-617670a8f3de None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:46:59.244346 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e6267b4-e0af-41df-8218-617670a8f3de None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:46:59.244614 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e6267b4-e0af-41df-8218-617670a8f3de None admin] PATH_INFO: `/v3/services` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:46:59.245239 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e6267b4-e0af-41df-8218-617670a8f3de None admin] RBAC: Adding query filter params (name=placement) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:46:59.245560 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e6267b4-e0af-41df-8218-617670a8f3de None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:46:59.246923 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e6267b4-e0af-41df-8218-617670a8f3de None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:46:59.251864 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 650/1291] 10.222.0.27 () {56 vars in 1116 bytes} [Tue Aug 30 01:46:59 2022] GET /identity/v3/services?name=placement => generated 126 bytes in 142 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:46:59.258438 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bd3031b-841a-4c30-a2a6-03a572c9ce12 None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:46:59.355382 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bd3031b-841a-4c30-a2a6-03a572c9ce12 None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:46:59.388910 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d40590d0-68e3-4af5-aef4-90a62a298e6f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:46:59.389526 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d40590d0-68e3-4af5-aef4-90a62a298e6f None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:46:59.389848 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d40590d0-68e3-4af5-aef4-90a62a298e6f None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:46:59.390127 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d40590d0-68e3-4af5-aef4-90a62a298e6f None admin] PATH_INFO: `/v3/services` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:46:59.390749 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d40590d0-68e3-4af5-aef4-90a62a298e6f None admin] RBAC: Adding query filter params (type=placement) {{(pid=81578) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:46:59.391046 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d40590d0-68e3-4af5-aef4-90a62a298e6f None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:46:59.392394 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d40590d0-68e3-4af5-aef4-90a62a298e6f None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:46:59.396760 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 642/1292] 10.222.0.27 () {56 vars in 1116 bytes} [Tue Aug 30 01:46:59 2022] GET /identity/v3/services?type=placement => generated 126 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:47:01.330624 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39fba01a-5da1-4daa-bc23-4d9834e7451d None None] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:47:01.331242 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39fba01a-5da1-4daa-bc23-4d9834e7451d None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:47:01.331242 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39fba01a-5da1-4daa-bc23-4d9834e7451d None None] PATH_INFO: `/` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:47:01.332536 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 651/1293] 10.222.0.27 () {54 vars in 927 bytes} [Tue Aug 30 01:47:01 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 01:47:01.337985 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bf3bf89-e635-47f8-9fd7-54356311ec34 None None] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:47:01.338455 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bf3bf89-e635-47f8-9fd7-54356311ec34 None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:47:01.338551 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bf3bf89-e635-47f8-9fd7-54356311ec34 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:47:01.357790 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.auth.core [None req-6bf3bf89-e635-47f8-9fd7-54356311ec34 None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81578) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 01:47:01.359012 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-6bf3bf89-e635-47f8-9fd7-54356311ec34 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:47:01.389115 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:47:01.429679 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 643/1294] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Aug 30 01:47:01 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 30 01:47:01.706135 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39fba01a-5da1-4daa-bc23-4d9834e7451d None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:47:01.739452 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:47:01.810051 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39fba01a-5da1-4daa-bc23-4d9834e7451d None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:47:01.845594 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbed75e1-b6cd-48fe-992a-1a959a83a5b0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:47:01.846095 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbed75e1-b6cd-48fe-992a-1a959a83a5b0 None admin] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:47:01.846373 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbed75e1-b6cd-48fe-992a-1a959a83a5b0 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:47:01.846645 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbed75e1-b6cd-48fe-992a-1a959a83a5b0 None admin] PATH_INFO: `/v3/services` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:47:01.847292 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbed75e1-b6cd-48fe-992a-1a959a83a5b0 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:47:01.849244 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbed75e1-b6cd-48fe-992a-1a959a83a5b0 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:47:01.960744 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 652/1295] 10.222.0.27 () {60 vars in 1141 bytes} [Tue Aug 30 01:47:01 2022] POST /identity/v3/services => generated 246 bytes in 255 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 01:47:03.913582 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8704ff66-eb9c-4ecf-a039-fe50b38e1526 None None] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:47:03.914116 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8704ff66-eb9c-4ecf-a039-fe50b38e1526 None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:47:03.914116 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8704ff66-eb9c-4ecf-a039-fe50b38e1526 None None] PATH_INFO: `/` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:47:03.915488 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 644/1296] 10.222.0.27 () {54 vars in 927 bytes} [Tue Aug 30 01:47:03 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 01:47:03.920762 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-647d8af4-dff7-4e14-8b66-a8c3a16a4023 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:47:03.924015 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-647d8af4-dff7-4e14-8b66-a8c3a16a4023 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:47:03.924015 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-647d8af4-dff7-4e14-8b66-a8c3a16a4023 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:47:03.939397 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-647d8af4-dff7-4e14-8b66-a8c3a16a4023 None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 01:47:03.940597 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-647d8af4-dff7-4e14-8b66-a8c3a16a4023 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:47:04.016239 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 653/1297] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Aug 30 01:47:03 2022] POST /identity/v3/auth/tokens => generated 2221 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 30 01:47:04.293759 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8704ff66-eb9c-4ecf-a039-fe50b38e1526 None None] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:47:04.398739 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8704ff66-eb9c-4ecf-a039-fe50b38e1526 None None] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:47:04.433821 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c1cfea5-6739-4ec8-b82a-d2bf3f821014 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:47:04.434452 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c1cfea5-6739-4ec8-b82a-d2bf3f821014 None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:47:04.434732 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c1cfea5-6739-4ec8-b82a-d2bf3f821014 None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:47:04.435001 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c1cfea5-6739-4ec8-b82a-d2bf3f821014 None admin] PATH_INFO: `/v3/services/placement` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:47:04.435577 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c1cfea5-6739-4ec8-b82a-d2bf3f821014 None admin] RBAC: Authorizing `identity:get_service(service_id=placement)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:47:04.436928 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c1cfea5-6739-4ec8-b82a-d2bf3f821014 None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:47:04.445974 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application [None req-6c1cfea5-6739-4ec8-b82a-d2bf3f821014 None admin] Could not find service: placement.: keystone.exception.ServiceNotFound: Could not find service: placement. Aug 30 01:47:04.445974 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 01:47:04.445974 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Aug 30 01:47:04.445974 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Aug 30 01:47:04.445974 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Aug 30 01:47:04.445974 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Aug 30 01:47:04.445974 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Aug 30 01:47:04.445974 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 30 01:47:04.445974 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Aug 30 01:47:04.445974 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application Aug 30 01:47:04.445974 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 30 01:47:04.445974 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application Aug 30 01:47:04.445974 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 01:47:04.445974 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 01:47:04.445974 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 01:47:04.445974 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 01:47:04.445974 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 01:47:04.447205 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 01:47:04.447205 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 01:47:04.447205 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 01:47:04.447205 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 01:47:04.447205 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 01:47:04.447205 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 01:47:04.447205 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Aug 30 01:47:04.447205 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._get_service(service_id) Aug 30 01:47:04.447205 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Aug 30 01:47:04.447205 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Aug 30 01:47:04.447205 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 01:47:04.447205 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 01:47:04.447205 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 01:47:04.447205 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 01:47:04.447205 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 01:47:04.447205 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 01:47:04.448236 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 01:47:04.448236 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application with Lock( Aug 30 01:47:04.448236 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 01:47:04.448236 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._enter() Aug 30 01:47:04.448236 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 01:47:04.448236 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 01:47:04.448236 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 01:47:04.448236 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.creator() Aug 30 01:47:04.448236 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 01:47:04.448236 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application created_value = creator( Aug 30 01:47:04.448236 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Aug 30 01:47:04.448236 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 30 01:47:04.448236 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Aug 30 01:47:04.448236 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application Aug 30 01:47:04.448236 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 645/1298] 10.222.0.27 () {56 vars in 1117 bytes} [Tue Aug 30 01:47:04 2022] GET /identity/v3/services/placement => generated 90 bytes in 154 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 01:47:04.453115 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-647d8af4-dff7-4e14-8b66-a8c3a16a4023 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:47:04.472239 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:47:04.554748 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-647d8af4-dff7-4e14-8b66-a8c3a16a4023 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:47:04.590211 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7101511-1adc-4bd9-a2f7-4451f1f7b799 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:47:04.590818 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7101511-1adc-4bd9-a2f7-4451f1f7b799 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:47:04.591094 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7101511-1adc-4bd9-a2f7-4451f1f7b799 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:47:04.591364 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7101511-1adc-4bd9-a2f7-4451f1f7b799 None admin] PATH_INFO: `/v3/services` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:47:04.592080 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7101511-1adc-4bd9-a2f7-4451f1f7b799 None admin] RBAC: Adding query filter params (name=placement) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:47:04.592381 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7101511-1adc-4bd9-a2f7-4451f1f7b799 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:47:04.593777 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7101511-1adc-4bd9-a2f7-4451f1f7b799 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:47:04.600426 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 654/1299] 10.222.0.27 () {56 vars in 1116 bytes} [Tue Aug 30 01:47:04 2022] GET /identity/v3/services?name=placement => generated 358 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:47:04.606239 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c1cfea5-6739-4ec8-b82a-d2bf3f821014 None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:47:04.638223 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:47:04.705218 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c1cfea5-6739-4ec8-b82a-d2bf3f821014 None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:47:04.741742 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6282ffc8-89cb-4141-9abf-21c127fb22c0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:47:04.742347 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6282ffc8-89cb-4141-9abf-21c127fb22c0 None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:47:04.742626 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6282ffc8-89cb-4141-9abf-21c127fb22c0 None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:47:04.742895 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6282ffc8-89cb-4141-9abf-21c127fb22c0 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:47:04.743737 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6282ffc8-89cb-4141-9abf-21c127fb22c0 None admin] RBAC: Adding query filter params (interface=public, service_id=8599060065a94897bbec0958c484390e, region_id=RegionOne) {{(pid=81578) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:47:04.744044 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6282ffc8-89cb-4141-9abf-21c127fb22c0 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:47:04.745429 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6282ffc8-89cb-4141-9abf-21c127fb22c0 None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:47:04.750988 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 646/1300] 10.222.0.27 () {56 vars in 1251 bytes} [Tue Aug 30 01:47:04 2022] GET /identity/v3/endpoints?service_id=8599060065a94897bbec0958c484390e&interface=public®ion_id=RegionOne => generated 194 bytes in 145 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:47:04.756848 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7101511-1adc-4bd9-a2f7-4451f1f7b799 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:47:04.838375 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:47:04.858845 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7101511-1adc-4bd9-a2f7-4451f1f7b799 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:47:04.900315 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9a1aaf2-3f6e-4df2-a51d-b88db388e238 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:47:04.900961 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9a1aaf2-3f6e-4df2-a51d-b88db388e238 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:47:04.901240 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9a1aaf2-3f6e-4df2-a51d-b88db388e238 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:47:04.901537 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9a1aaf2-3f6e-4df2-a51d-b88db388e238 None admin] PATH_INFO: `/v3/services` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:47:04.902142 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9a1aaf2-3f6e-4df2-a51d-b88db388e238 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:47:04.903513 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9a1aaf2-3f6e-4df2-a51d-b88db388e238 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:47:04.911848 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 655/1301] 10.222.0.27 () {56 vars in 1087 bytes} [Tue Aug 30 01:47:04 2022] GET /identity/v3/services => generated 2191 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 01:47:06.820487 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea3a777c-33cf-4a99-8a4b-293f57022ebd None None] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:47:06.821037 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea3a777c-33cf-4a99-8a4b-293f57022ebd None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:47:06.821161 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea3a777c-33cf-4a99-8a4b-293f57022ebd None None] PATH_INFO: `/` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:47:06.822492 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 647/1302] 10.222.0.27 () {54 vars in 927 bytes} [Tue Aug 30 01:47:06 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 01:47:06.827791 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcf93f05-90b4-47ab-99ec-d708c32ec397 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:47:06.828262 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcf93f05-90b4-47ab-99ec-d708c32ec397 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:47:06.828350 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcf93f05-90b4-47ab-99ec-d708c32ec397 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:47:06.846139 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-dcf93f05-90b4-47ab-99ec-d708c32ec397 None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 01:47:06.847321 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-dcf93f05-90b4-47ab-99ec-d708c32ec397 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:47:06.929228 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 656/1303] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Aug 30 01:47:06 2022] POST /identity/v3/auth/tokens => generated 2221 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 30 01:47:07.201030 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea3a777c-33cf-4a99-8a4b-293f57022ebd None None] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:47:07.298466 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:47:07.305230 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea3a777c-33cf-4a99-8a4b-293f57022ebd None None] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:47:07.338011 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb853fae-7a66-4b51-a74d-848e2fc9c991 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:47:07.338626 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb853fae-7a66-4b51-a74d-848e2fc9c991 None admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:47:07.338902 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb853fae-7a66-4b51-a74d-848e2fc9c991 None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:47:07.339171 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb853fae-7a66-4b51-a74d-848e2fc9c991 None admin] PATH_INFO: `/v3/services/placement` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:47:07.339745 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb853fae-7a66-4b51-a74d-848e2fc9c991 None admin] RBAC: Authorizing `identity:get_service(service_id=placement)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:47:07.341096 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb853fae-7a66-4b51-a74d-848e2fc9c991 None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:47:07.351257 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application [None req-eb853fae-7a66-4b51-a74d-848e2fc9c991 None admin] Could not find service: placement.: keystone.exception.ServiceNotFound: Could not find service: placement. Aug 30 01:47:07.351257 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 01:47:07.351257 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Aug 30 01:47:07.351257 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Aug 30 01:47:07.351257 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Aug 30 01:47:07.351257 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Aug 30 01:47:07.351257 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Aug 30 01:47:07.351257 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 30 01:47:07.351257 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Aug 30 01:47:07.351257 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application Aug 30 01:47:07.351257 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 30 01:47:07.351257 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application Aug 30 01:47:07.351257 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 30 01:47:07.351257 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 30 01:47:07.351257 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 30 01:47:07.351257 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 30 01:47:07.351257 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 30 01:47:07.352293 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 30 01:47:07.352293 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 30 01:47:07.352293 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 30 01:47:07.352293 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 30 01:47:07.352293 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 30 01:47:07.352293 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 30 01:47:07.352293 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Aug 30 01:47:07.352293 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._get_service(service_id) Aug 30 01:47:07.352293 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Aug 30 01:47:07.352293 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Aug 30 01:47:07.352293 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 30 01:47:07.352293 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 30 01:47:07.352293 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 30 01:47:07.352293 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 30 01:47:07.352293 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 30 01:47:07.352293 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.get_or_create( Aug 30 01:47:07.353211 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 30 01:47:07.353211 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application with Lock( Aug 30 01:47:07.353211 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 30 01:47:07.353211 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self._enter() Aug 30 01:47:07.353211 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 30 01:47:07.353211 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 30 01:47:07.353211 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 30 01:47:07.353211 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application return self.creator() Aug 30 01:47:07.353211 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 30 01:47:07.353211 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application created_value = creator( Aug 30 01:47:07.353211 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Aug 30 01:47:07.353211 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 30 01:47:07.353211 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Aug 30 01:47:07.353211 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: ERROR keystone.server.flask.application Aug 30 01:47:07.353211 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 648/1304] 10.222.0.27 () {56 vars in 1117 bytes} [Tue Aug 30 01:47:07 2022] GET /identity/v3/services/placement => generated 90 bytes in 152 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 30 01:47:07.358548 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcf93f05-90b4-47ab-99ec-d708c32ec397 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:47:07.433782 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:47:07.456298 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcf93f05-90b4-47ab-99ec-d708c32ec397 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:47:07.490489 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4abdee8a-d0ae-464f-b441-d06c62d52051 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:47:07.491083 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4abdee8a-d0ae-464f-b441-d06c62d52051 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:47:07.491357 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4abdee8a-d0ae-464f-b441-d06c62d52051 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:47:07.491624 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4abdee8a-d0ae-464f-b441-d06c62d52051 None admin] PATH_INFO: `/v3/services` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:47:07.492248 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4abdee8a-d0ae-464f-b441-d06c62d52051 None admin] RBAC: Adding query filter params (name=placement) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:47:07.492544 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4abdee8a-d0ae-464f-b441-d06c62d52051 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:47:07.493948 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4abdee8a-d0ae-464f-b441-d06c62d52051 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:47:07.499035 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 657/1305] 10.222.0.27 () {56 vars in 1116 bytes} [Tue Aug 30 01:47:07 2022] GET /identity/v3/services?name=placement => generated 358 bytes in 141 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:47:07.505538 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb853fae-7a66-4b51-a74d-848e2fc9c991 None admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:47:07.606298 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb853fae-7a66-4b51-a74d-848e2fc9c991 None admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:47:07.640572 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f3d7238-c133-41a1-b83e-4579e9b0100c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:47:07.641169 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f3d7238-c133-41a1-b83e-4579e9b0100c None admin] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:47:07.641449 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f3d7238-c133-41a1-b83e-4579e9b0100c None admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:47:07.641742 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f3d7238-c133-41a1-b83e-4579e9b0100c None admin] PATH_INFO: `/v3/endpoints` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:47:07.642400 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f3d7238-c133-41a1-b83e-4579e9b0100c None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:47:07.643748 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f3d7238-c133-41a1-b83e-4579e9b0100c None admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:47:07.750062 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 649/1306] 10.222.0.27 () {60 vars in 1144 bytes} [Tue Aug 30 01:47:07 2022] POST /identity/v3/endpoints => generated 331 bytes in 245 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 01:47:17.496108 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=30 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:47:17.745292 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=91,keystone:INSERT=1 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:53:32.745725 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ddc85a0-ad99-416d-aa9e-730c80a37313 None None] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:53:32.746720 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ddc85a0-ad99-416d-aa9e-730c80a37313 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:53:32.747209 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ddc85a0-ad99-416d-aa9e-730c80a37313 None None] PATH_INFO: `/` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:53:32.754694 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0aa9c7a-b8df-457d-a180-6f2bd7867126 None None] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:53:32.755558 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0aa9c7a-b8df-457d-a180-6f2bd7867126 None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:53:32.756130 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0aa9c7a-b8df-457d-a180-6f2bd7867126 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:53:32.761025 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 658/1307] 10.222.0.27 () {54 vars in 927 bytes} [Tue Aug 30 01:53:32 2022] GET /identity => generated 270 bytes in 8 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 01:53:32.858418 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.auth.core [None req-d0aa9c7a-b8df-457d-a180-6f2bd7867126 None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81578) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 01:53:32.859994 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-d0aa9c7a-b8df-457d-a180-6f2bd7867126 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:53:33.008330 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 650/1308] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Aug 30 01:53:32 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 254 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 01:53:33.344201 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-beae6742-17e0-4e8e-beee-5efbf45128ae None None] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:53:33.344519 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-beae6742-17e0-4e8e-beee-5efbf45128ae None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:53:33.344693 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-beae6742-17e0-4e8e-beee-5efbf45128ae None None] PATH_INFO: `/` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:53:33.346786 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 659/1309] 10.222.0.27 () {54 vars in 965 bytes} [Tue Aug 30 01:53:33 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 01:53:33.354564 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d1e2fe6-6d25-4a27-a53e-7df0e938aba1 None None] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:53:33.355245 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d1e2fe6-6d25-4a27-a53e-7df0e938aba1 None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:53:33.355758 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d1e2fe6-6d25-4a27-a53e-7df0e938aba1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:53:33.438739 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.auth.core [None req-6d1e2fe6-6d25-4a27-a53e-7df0e938aba1 None None] MFA Rules not processed for user `f3ea1580c56f4814a92bbb2acf542905`. Rule list: `[]` (Enabled: `True`). {{(pid=81578) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 01:53:33.440430 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-6d1e2fe6-6d25-4a27-a53e-7df0e938aba1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:53:33.462345 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:53:33.521840 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 651/1310] 10.222.0.27 () {58 vars in 1062 bytes} [Tue Aug 30 01:53:33 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 169 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 01:53:33.530860 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-beae6742-17e0-4e8e-beee-5efbf45128ae None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:53:33.532704 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-beae6742-17e0-4e8e-beee-5efbf45128ae None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:53:33.643613 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-beae6742-17e0-4e8e-beee-5efbf45128ae None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:53:33.721526 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bd4e193-5440-4f99-88c5-f8739d66584a service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f3ea1580c56f4814a92bbb2acf542905', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4562f3951fbe465280073e60a11e44e2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:53:33.721526 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bd4e193-5440-4f99-88c5-f8739d66584a service neutron] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:53:33.721779 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bd4e193-5440-4f99-88c5-f8739d66584a service neutron] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:53:33.722106 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bd4e193-5440-4f99-88c5-f8739d66584a service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:53:33.723956 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-4bd4e193-5440-4f99-88c5-f8739d66584a service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:53:33.779804 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:53:33.839863 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4bd4e193-5440-4f99-88c5-f8739d66584a service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:53:33.841323 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4bd4e193-5440-4f99-88c5-f8739d66584a service neutron] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:53:33.865205 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 660/1311] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Aug 30 01:53:33 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 335 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 01:53:36.070958 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90d41214-a6cd-462f-80f7-985aaba97f6d None None] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:53:36.071497 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90d41214-a6cd-462f-80f7-985aaba97f6d None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:53:36.071497 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90d41214-a6cd-462f-80f7-985aaba97f6d None None] PATH_INFO: `/` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:53:36.072834 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 652/1312] 10.222.0.27 () {54 vars in 927 bytes} [Tue Aug 30 01:53:36 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 01:53:36.078004 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab037f0d-39bd-4fb5-a9da-9b4a2703fd3c None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:53:36.078392 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab037f0d-39bd-4fb5-a9da-9b4a2703fd3c None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:53:36.078548 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab037f0d-39bd-4fb5-a9da-9b4a2703fd3c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:53:36.153503 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-ab037f0d-39bd-4fb5-a9da-9b4a2703fd3c None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 01:53:36.154704 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-ab037f0d-39bd-4fb5-a9da-9b4a2703fd3c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:53:36.264630 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 661/1313] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Aug 30 01:53:36 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 188 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 01:53:36.575018 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5631f6a-aaf8-4032-aaa4-0df2b04c943e None None] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:53:36.575309 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5631f6a-aaf8-4032-aaa4-0df2b04c943e None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:53:36.575586 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5631f6a-aaf8-4032-aaa4-0df2b04c943e None None] PATH_INFO: `/` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:53:36.576931 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 653/1314] 10.222.0.27 () {54 vars in 965 bytes} [Tue Aug 30 01:53:36 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 01:53:36.583210 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56f74b65-df1f-4a89-9a0a-2b82b6a456a5 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:53:36.583565 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56f74b65-df1f-4a89-9a0a-2b82b6a456a5 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:53:36.583905 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56f74b65-df1f-4a89-9a0a-2b82b6a456a5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:53:36.596546 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:53:36.661167 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-56f74b65-df1f-4a89-9a0a-2b82b6a456a5 None None] MFA Rules not processed for user `f3ea1580c56f4814a92bbb2acf542905`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 01:53:36.662421 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-56f74b65-df1f-4a89-9a0a-2b82b6a456a5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:53:36.737856 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 662/1315] 10.222.0.27 () {58 vars in 1062 bytes} [Tue Aug 30 01:53:36 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 156 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 01:53:36.746992 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5631f6a-aaf8-4032-aaa4-0df2b04c943e None None] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:53:36.748518 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-f5631f6a-aaf8-4032-aaa4-0df2b04c943e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:53:36.849946 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5631f6a-aaf8-4032-aaa4-0df2b04c943e None None] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:53:36.917930 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:53:36.920442 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c900839-dc77-45cf-a06a-d3aa2ab54bfa service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f3ea1580c56f4814a92bbb2acf542905', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4562f3951fbe465280073e60a11e44e2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:53:36.921032 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c900839-dc77-45cf-a06a-d3aa2ab54bfa service neutron] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:53:36.921309 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c900839-dc77-45cf-a06a-d3aa2ab54bfa service neutron] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:53:36.921607 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c900839-dc77-45cf-a06a-d3aa2ab54bfa service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:53:36.923185 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-8c900839-dc77-45cf-a06a-d3aa2ab54bfa service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:53:37.029077 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c900839-dc77-45cf-a06a-d3aa2ab54bfa service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:53:37.030533 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c900839-dc77-45cf-a06a-d3aa2ab54bfa service neutron] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:53:37.048430 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 654/1316] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Aug 30 01:53:36 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 302 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 30 01:53:39.722068 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ec7a218-2361-49b8-9703-732067dcacd0 None None] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:53:39.722068 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ec7a218-2361-49b8-9703-732067dcacd0 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:53:39.722068 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ec7a218-2361-49b8-9703-732067dcacd0 None None] PATH_INFO: `/` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:53:39.723621 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 663/1317] 10.222.0.27 () {54 vars in 927 bytes} [Tue Aug 30 01:53:39 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 01:53:39.728880 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c03a458-5e57-461c-92e1-4da6e04ccd3b None None] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:53:39.729328 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c03a458-5e57-461c-92e1-4da6e04ccd3b None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:53:39.729422 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c03a458-5e57-461c-92e1-4da6e04ccd3b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:53:39.795062 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.auth.core [None req-5c03a458-5e57-461c-92e1-4da6e04ccd3b None None] MFA Rules not processed for user `fac663577ae547588ada9d16b2fd6f5f`. Rule list: `[]` (Enabled: `True`). {{(pid=81578) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 01:53:39.796261 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-5c03a458-5e57-461c-92e1-4da6e04ccd3b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:53:39.846494 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:53:39.889809 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 655/1318] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Aug 30 01:53:39 2022] POST /identity/v3/auth/tokens => generated 3243 bytes in 162 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 01:53:40.178746 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ec7a218-2361-49b8-9703-732067dcacd0 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:53:40.180275 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-5ec7a218-2361-49b8-9703-732067dcacd0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:53:40.282371 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ec7a218-2361-49b8-9703-732067dcacd0 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:53:40.285183 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:53:40.355262 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab37cbc8-df57-4ee6-8e79-3d3ef87e340a service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f3ea1580c56f4814a92bbb2acf542905', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4562f3951fbe465280073e60a11e44e2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:53:40.355858 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab37cbc8-df57-4ee6-8e79-3d3ef87e340a service neutron] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:53:40.356137 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab37cbc8-df57-4ee6-8e79-3d3ef87e340a service neutron] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:53:40.356408 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab37cbc8-df57-4ee6-8e79-3d3ef87e340a service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:53:40.358021 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-ab37cbc8-df57-4ee6-8e79-3d3ef87e340a service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:53:40.464685 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab37cbc8-df57-4ee6-8e79-3d3ef87e340a service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:53:40.466073 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab37cbc8-df57-4ee6-8e79-3d3ef87e340a service neutron] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:53:40.485429 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 664/1319] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Aug 30 01:53:40 2022] GET /identity/v3/auth/tokens => generated 3243 bytes in 307 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 01:53:43.172909 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e42a0373-3926-440f-8a26-46f64abc37f5 None None] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:53:43.173488 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e42a0373-3926-440f-8a26-46f64abc37f5 None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:53:43.173488 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e42a0373-3926-440f-8a26-46f64abc37f5 None None] PATH_INFO: `/` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:53:43.175022 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 656/1320] 10.222.0.27 () {54 vars in 927 bytes} [Tue Aug 30 01:53:43 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 01:53:43.180250 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6caf9fee-2eb5-48ce-ab39-31de3f8dd1ae None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:53:43.180652 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6caf9fee-2eb5-48ce-ab39-31de3f8dd1ae None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:53:43.180790 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6caf9fee-2eb5-48ce-ab39-31de3f8dd1ae None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:53:43.221360 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:53:43.246383 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-6caf9fee-2eb5-48ce-ab39-31de3f8dd1ae None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 01:53:43.247633 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-6caf9fee-2eb5-48ce-ab39-31de3f8dd1ae None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:53:43.345176 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 665/1321] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Aug 30 01:53:43 2022] POST /identity/v3/auth/tokens => generated 3238 bytes in 166 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 01:53:43.631995 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e42a0373-3926-440f-8a26-46f64abc37f5 None None] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:53:43.633550 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-e42a0373-3926-440f-8a26-46f64abc37f5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:53:43.733877 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e42a0373-3926-440f-8a26-46f64abc37f5 None None] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:53:43.805886 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c3da8ef-49fc-4170-a816-6042f7eef948 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f3ea1580c56f4814a92bbb2acf542905', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4562f3951fbe465280073e60a11e44e2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:53:43.806487 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c3da8ef-49fc-4170-a816-6042f7eef948 service neutron] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:53:43.806759 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c3da8ef-49fc-4170-a816-6042f7eef948 service neutron] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:53:43.807021 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c3da8ef-49fc-4170-a816-6042f7eef948 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:53:43.808601 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-3c3da8ef-49fc-4170-a816-6042f7eef948 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:53:43.816541 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:53:43.914816 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c3da8ef-49fc-4170-a816-6042f7eef948 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:53:43.916159 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c3da8ef-49fc-4170-a816-6042f7eef948 service neutron] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:53:43.933857 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 657/1322] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Aug 30 01:53:43 2022] GET /identity/v3/auth/tokens => generated 3238 bytes in 302 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 01:53:46.988868 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06de5dbf-d390-4287-b322-298db44abb69 None None] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:53:46.989376 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06de5dbf-d390-4287-b322-298db44abb69 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:53:46.989376 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06de5dbf-d390-4287-b322-298db44abb69 None None] PATH_INFO: `/` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:53:46.990712 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 666/1323] 10.222.0.27 () {54 vars in 927 bytes} [Tue Aug 30 01:53:46 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 01:53:46.995868 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea89131d-51b6-4637-acc4-bd5cf57fe608 None None] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:53:46.996303 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea89131d-51b6-4637-acc4-bd5cf57fe608 None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:53:46.996492 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea89131d-51b6-4637-acc4-bd5cf57fe608 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:53:47.061920 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.auth.core [None req-ea89131d-51b6-4637-acc4-bd5cf57fe608 None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81578) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 01:53:47.063145 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-ea89131d-51b6-4637-acc4-bd5cf57fe608 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:53:47.121726 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:53:47.161167 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 658/1324] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Aug 30 01:53:46 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 166 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 01:53:47.447170 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06de5dbf-d390-4287-b322-298db44abb69 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:53:47.448808 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-06de5dbf-d390-4287-b322-298db44abb69 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:53:47.560167 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06de5dbf-d390-4287-b322-298db44abb69 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:53:47.572962 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:53:47.635740 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24ff29ec-8dc8-4cf0-adc1-2e66cd168d9e service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f3ea1580c56f4814a92bbb2acf542905', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4562f3951fbe465280073e60a11e44e2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:53:47.636343 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24ff29ec-8dc8-4cf0-adc1-2e66cd168d9e service neutron] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:53:47.636618 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24ff29ec-8dc8-4cf0-adc1-2e66cd168d9e service neutron] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:53:47.636884 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24ff29ec-8dc8-4cf0-adc1-2e66cd168d9e service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:53:47.638481 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-24ff29ec-8dc8-4cf0-adc1-2e66cd168d9e service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:53:47.750353 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24ff29ec-8dc8-4cf0-adc1-2e66cd168d9e service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:53:47.752451 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24ff29ec-8dc8-4cf0-adc1-2e66cd168d9e service neutron] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:53:47.772044 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 667/1325] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Aug 30 01:53:47 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 325 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 01:53:49.756183 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-168bc4dd-162e-42f4-84d6-83b43ac66370 None None] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:53:49.756744 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-168bc4dd-162e-42f4-84d6-83b43ac66370 None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:53:49.756850 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-168bc4dd-162e-42f4-84d6-83b43ac66370 None None] PATH_INFO: `/` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:53:49.758197 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 659/1326] 10.222.0.27 () {54 vars in 927 bytes} [Tue Aug 30 01:53:49 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 01:53:49.765729 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88775e83-dca0-4994-b792-ecb0233fb227 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:53:49.766211 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88775e83-dca0-4994-b792-ecb0233fb227 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:53:49.766308 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88775e83-dca0-4994-b792-ecb0233fb227 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:53:49.817068 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:53:49.836159 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-88775e83-dca0-4994-b792-ecb0233fb227 None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 01:53:49.837376 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-88775e83-dca0-4994-b792-ecb0233fb227 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:53:49.935001 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 668/1327] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Aug 30 01:53:49 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 170 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 01:53:50.219396 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-168bc4dd-162e-42f4-84d6-83b43ac66370 None None] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:53:50.221695 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-168bc4dd-162e-42f4-84d6-83b43ac66370 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:53:50.325847 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-168bc4dd-162e-42f4-84d6-83b43ac66370 None None] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:53:50.401118 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8965ed7e-99c2-4c4b-b0ad-53150cec5578 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f3ea1580c56f4814a92bbb2acf542905', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4562f3951fbe465280073e60a11e44e2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:53:50.401746 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8965ed7e-99c2-4c4b-b0ad-53150cec5578 service neutron] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:53:50.402024 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8965ed7e-99c2-4c4b-b0ad-53150cec5578 service neutron] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:53:50.402289 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8965ed7e-99c2-4c4b-b0ad-53150cec5578 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:53:50.403952 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-8965ed7e-99c2-4c4b-b0ad-53150cec5578 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:53:50.431337 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:53:50.525570 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8965ed7e-99c2-4c4b-b0ad-53150cec5578 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:53:50.527002 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8965ed7e-99c2-4c4b-b0ad-53150cec5578 service neutron] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:53:50.545572 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 660/1328] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Aug 30 01:53:50 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 327 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 01:53:52.584823 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72666c55-46cf-4e2c-a95c-c1b7e754a35b None None] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:53:52.585285 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72666c55-46cf-4e2c-a95c-c1b7e754a35b None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:53:52.585495 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72666c55-46cf-4e2c-a95c-c1b7e754a35b None None] PATH_INFO: `/` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:53:52.586729 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 669/1329] 10.222.0.27 () {54 vars in 927 bytes} [Tue Aug 30 01:53:52 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 01:53:52.592076 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5187838-ce10-4391-adfb-3cb714fe4966 None None] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:53:52.592512 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5187838-ce10-4391-adfb-3cb714fe4966 None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:53:52.592754 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5187838-ce10-4391-adfb-3cb714fe4966 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:53:52.665347 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.auth.core [None req-b5187838-ce10-4391-adfb-3cb714fe4966 None None] MFA Rules not processed for user `fac663577ae547588ada9d16b2fd6f5f`. Rule list: `[]` (Enabled: `True`). {{(pid=81578) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 01:53:52.666595 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-b5187838-ce10-4391-adfb-3cb714fe4966 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:53:52.733115 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:53:52.766531 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 661/1330] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Aug 30 01:53:52 2022] POST /identity/v3/auth/tokens => generated 3243 bytes in 175 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 01:53:53.057799 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72666c55-46cf-4e2c-a95c-c1b7e754a35b None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:53:53.059318 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-72666c55-46cf-4e2c-a95c-c1b7e754a35b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:53:53.176442 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72666c55-46cf-4e2c-a95c-c1b7e754a35b None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:53:53.196111 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:53:53.249217 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04bbf48d-3bbf-43b6-bd55-260a59bfe259 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f3ea1580c56f4814a92bbb2acf542905', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4562f3951fbe465280073e60a11e44e2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:53:53.249853 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04bbf48d-3bbf-43b6-bd55-260a59bfe259 service neutron] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:53:53.250140 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04bbf48d-3bbf-43b6-bd55-260a59bfe259 service neutron] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:53:53.250614 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04bbf48d-3bbf-43b6-bd55-260a59bfe259 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:53:53.253074 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-04bbf48d-3bbf-43b6-bd55-260a59bfe259 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:53:53.361845 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04bbf48d-3bbf-43b6-bd55-260a59bfe259 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:53:53.363216 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04bbf48d-3bbf-43b6-bd55-260a59bfe259 service neutron] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:53:53.381171 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 670/1331] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Aug 30 01:53:53 2022] GET /identity/v3/auth/tokens => generated 3243 bytes in 324 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 01:53:55.660092 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14928055-d7e4-4e6f-9d7a-f348d9182e1f None None] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:53:55.660654 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14928055-d7e4-4e6f-9d7a-f348d9182e1f None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:53:55.660654 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14928055-d7e4-4e6f-9d7a-f348d9182e1f None None] PATH_INFO: `/` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:53:55.662000 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 662/1332] 10.222.0.27 () {54 vars in 927 bytes} [Tue Aug 30 01:53:55 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 01:53:55.667076 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2aa61269-f74c-4f28-b0b5-c759ec37054f None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:53:55.667496 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2aa61269-f74c-4f28-b0b5-c759ec37054f None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:53:55.667622 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2aa61269-f74c-4f28-b0b5-c759ec37054f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:53:55.732085 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:53:55.734671 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-2aa61269-f74c-4f28-b0b5-c759ec37054f None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 01:53:55.736572 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-2aa61269-f74c-4f28-b0b5-c759ec37054f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:53:55.832204 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 671/1333] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Aug 30 01:53:55 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 166 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 01:53:56.114635 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14928055-d7e4-4e6f-9d7a-f348d9182e1f None None] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:53:56.116172 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-14928055-d7e4-4e6f-9d7a-f348d9182e1f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:53:56.226564 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14928055-d7e4-4e6f-9d7a-f348d9182e1f None None] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:53:56.297131 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba669a60-bbd6-458e-ba1a-8a37612231c7 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f3ea1580c56f4814a92bbb2acf542905', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4562f3951fbe465280073e60a11e44e2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:53:56.297788 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba669a60-bbd6-458e-ba1a-8a37612231c7 service neutron] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:53:56.298145 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba669a60-bbd6-458e-ba1a-8a37612231c7 service neutron] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:53:56.298418 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba669a60-bbd6-458e-ba1a-8a37612231c7 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:53:56.299986 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-ba669a60-bbd6-458e-ba1a-8a37612231c7 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:53:56.326087 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:53:56.409602 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba669a60-bbd6-458e-ba1a-8a37612231c7 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:53:56.410956 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba669a60-bbd6-458e-ba1a-8a37612231c7 service neutron] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:53:56.429284 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 663/1334] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Aug 30 01:53:56 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 315 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 01:53:58.418905 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f869d8d-8fe3-4d24-89ed-59d5315dc379 None None] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:53:58.419482 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f869d8d-8fe3-4d24-89ed-59d5315dc379 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:53:58.419482 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f869d8d-8fe3-4d24-89ed-59d5315dc379 None None] PATH_INFO: `/` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:53:58.420801 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 672/1335] 10.222.0.27 () {54 vars in 927 bytes} [Tue Aug 30 01:53:58 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 01:53:58.426944 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11db6c80-b3f4-4588-952a-9818de7ca2fb None None] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:53:58.427356 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11db6c80-b3f4-4588-952a-9818de7ca2fb None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:53:58.427514 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11db6c80-b3f4-4588-952a-9818de7ca2fb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:53:58.492958 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.auth.core [None req-11db6c80-b3f4-4588-952a-9818de7ca2fb None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81578) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 01:53:58.494181 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-11db6c80-b3f4-4588-952a-9818de7ca2fb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:53:58.574639 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:53:58.597702 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 664/1336] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Aug 30 01:53:58 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 172 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 01:53:58.881155 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f869d8d-8fe3-4d24-89ed-59d5315dc379 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:53:58.882739 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-0f869d8d-8fe3-4d24-89ed-59d5315dc379 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:53:58.983678 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f869d8d-8fe3-4d24-89ed-59d5315dc379 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:53:59.011695 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:53:59.057122 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0075f01a-f16f-4fa0-bc95-781ca3baa03e service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f3ea1580c56f4814a92bbb2acf542905', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4562f3951fbe465280073e60a11e44e2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:53:59.057781 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0075f01a-f16f-4fa0-bc95-781ca3baa03e service neutron] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:53:59.058055 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0075f01a-f16f-4fa0-bc95-781ca3baa03e service neutron] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:53:59.058318 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0075f01a-f16f-4fa0-bc95-781ca3baa03e service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:53:59.059916 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-0075f01a-f16f-4fa0-bc95-781ca3baa03e service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:53:59.164524 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0075f01a-f16f-4fa0-bc95-781ca3baa03e service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:53:59.166029 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0075f01a-f16f-4fa0-bc95-781ca3baa03e service neutron] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:53:59.184373 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 673/1337] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Aug 30 01:53:58 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 304 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 01:54:01.777830 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be4edd60-07e4-4ca4-b3d6-8b9f67bca535 None None] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:54:01.778389 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be4edd60-07e4-4ca4-b3d6-8b9f67bca535 None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:54:01.778389 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be4edd60-07e4-4ca4-b3d6-8b9f67bca535 None None] PATH_INFO: `/` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:54:01.779707 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 665/1338] 10.222.0.27 () {54 vars in 927 bytes} [Tue Aug 30 01:54:01 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 01:54:01.784840 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-915cefd5-9b48-4d2b-b329-a1845c83a205 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:54:01.785265 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-915cefd5-9b48-4d2b-b329-a1845c83a205 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:54:01.785382 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-915cefd5-9b48-4d2b-b329-a1845c83a205 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:54:01.852062 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-915cefd5-9b48-4d2b-b329-a1845c83a205 None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 01:54:01.853279 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-915cefd5-9b48-4d2b-b329-a1845c83a205 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:54:01.862358 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:54:01.948950 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 674/1339] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Aug 30 01:54:01 2022] POST /identity/v3/auth/tokens => generated 3238 bytes in 165 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 01:54:02.241628 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be4edd60-07e4-4ca4-b3d6-8b9f67bca535 None None] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:54:02.243221 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-be4edd60-07e4-4ca4-b3d6-8b9f67bca535 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:54:02.347812 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be4edd60-07e4-4ca4-b3d6-8b9f67bca535 None None] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:54:02.420735 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5967ad17-8d39-4af1-924f-4880b5a35822 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f3ea1580c56f4814a92bbb2acf542905', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4562f3951fbe465280073e60a11e44e2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:54:02.421270 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5967ad17-8d39-4af1-924f-4880b5a35822 service neutron] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:54:02.421577 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5967ad17-8d39-4af1-924f-4880b5a35822 service neutron] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:54:02.421848 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5967ad17-8d39-4af1-924f-4880b5a35822 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:54:02.424031 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-5967ad17-8d39-4af1-924f-4880b5a35822 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:54:02.462163 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:54:02.545043 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5967ad17-8d39-4af1-924f-4880b5a35822 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:54:02.546499 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5967ad17-8d39-4af1-924f-4880b5a35822 service neutron] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:54:02.565526 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 666/1340] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Aug 30 01:54:02 2022] GET /identity/v3/auth/tokens => generated 3238 bytes in 324 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 01:54:06.651692 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-599b26c3-b3d4-4dbf-8eff-63621c168a71 None None] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:54:06.652261 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-599b26c3-b3d4-4dbf-8eff-63621c168a71 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:54:06.652261 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-599b26c3-b3d4-4dbf-8eff-63621c168a71 None None] PATH_INFO: `/` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:54:06.653670 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 675/1341] 10.222.0.27 () {54 vars in 927 bytes} [Tue Aug 30 01:54:06 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 01:54:06.659946 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-753fd321-e1da-42a5-b050-e85523ff4d00 None None] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:54:06.660411 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-753fd321-e1da-42a5-b050-e85523ff4d00 None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:54:06.660575 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-753fd321-e1da-42a5-b050-e85523ff4d00 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:54:06.730763 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.auth.core [None req-753fd321-e1da-42a5-b050-e85523ff4d00 None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81578) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 01:54:06.731990 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-753fd321-e1da-42a5-b050-e85523ff4d00 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:54:06.819111 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:54:06.830740 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 667/1342] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Aug 30 01:54:06 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 172 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 01:54:07.115298 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-599b26c3-b3d4-4dbf-8eff-63621c168a71 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:54:07.116839 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-599b26c3-b3d4-4dbf-8eff-63621c168a71 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:54:07.225055 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-599b26c3-b3d4-4dbf-8eff-63621c168a71 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:54:07.263870 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:54:07.300878 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74d73c06-fde3-4d04-ba28-515c3a8a06eb service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f3ea1580c56f4814a92bbb2acf542905', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4562f3951fbe465280073e60a11e44e2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:54:07.301519 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74d73c06-fde3-4d04-ba28-515c3a8a06eb service neutron] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:54:07.301814 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74d73c06-fde3-4d04-ba28-515c3a8a06eb service neutron] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:54:07.302926 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74d73c06-fde3-4d04-ba28-515c3a8a06eb service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:54:07.304529 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-74d73c06-fde3-4d04-ba28-515c3a8a06eb service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:54:07.417249 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74d73c06-fde3-4d04-ba28-515c3a8a06eb service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:54:07.418641 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74d73c06-fde3-4d04-ba28-515c3a8a06eb service neutron] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:54:07.436902 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 676/1343] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Aug 30 01:54:07 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 322 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 01:54:10.346524 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f086cfc2-9659-485c-8eb6-098e6eabe502 None None] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:54:10.347329 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f086cfc2-9659-485c-8eb6-098e6eabe502 None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:54:10.347329 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f086cfc2-9659-485c-8eb6-098e6eabe502 None None] PATH_INFO: `/` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:54:10.348723 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 668/1344] 10.222.0.27 () {54 vars in 927 bytes} [Tue Aug 30 01:54:10 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 01:54:10.354055 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-880c40c1-da1d-4e6d-8057-e452b580cc73 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:54:10.354532 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-880c40c1-da1d-4e6d-8057-e452b580cc73 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:54:10.354741 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-880c40c1-da1d-4e6d-8057-e452b580cc73 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:54:10.420105 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-880c40c1-da1d-4e6d-8057-e452b580cc73 None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 01:54:10.421218 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-880c40c1-da1d-4e6d-8057-e452b580cc73 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:54:10.440617 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:54:10.519931 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 677/1345] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Aug 30 01:54:10 2022] POST /identity/v3/auth/tokens => generated 3238 bytes in 167 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 01:54:10.804033 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f086cfc2-9659-485c-8eb6-098e6eabe502 None None] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:54:10.805722 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-f086cfc2-9659-485c-8eb6-098e6eabe502 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:54:10.912817 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f086cfc2-9659-485c-8eb6-098e6eabe502 None None] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:54:10.987428 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68ec0861-e983-4238-b275-4d580448dd16 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f3ea1580c56f4814a92bbb2acf542905', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4562f3951fbe465280073e60a11e44e2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:54:10.987991 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68ec0861-e983-4238-b275-4d580448dd16 service neutron] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:54:10.988275 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68ec0861-e983-4238-b275-4d580448dd16 service neutron] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:54:10.988551 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68ec0861-e983-4238-b275-4d580448dd16 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:54:10.990212 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-68ec0861-e983-4238-b275-4d580448dd16 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:54:11.036482 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:54:11.105714 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68ec0861-e983-4238-b275-4d580448dd16 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:54:11.107173 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68ec0861-e983-4238-b275-4d580448dd16 service neutron] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:54:11.129202 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 669/1346] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Aug 30 01:54:10 2022] GET /identity/v3/auth/tokens => generated 3238 bytes in 326 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 01:54:15.292576 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c641b280-ff2b-48d5-b3d3-7fbcc3a89574 None None] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:54:15.293078 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c641b280-ff2b-48d5-b3d3-7fbcc3a89574 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:54:15.293176 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c641b280-ff2b-48d5-b3d3-7fbcc3a89574 None None] PATH_INFO: `/` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:54:15.294531 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 678/1347] 10.222.0.27 () {54 vars in 927 bytes} [Tue Aug 30 01:54:15 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 01:54:15.300533 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1a1c9f0-7276-4833-9bed-94d4cc22da44 None None] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:54:15.301022 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1a1c9f0-7276-4833-9bed-94d4cc22da44 None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:54:15.301104 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1a1c9f0-7276-4833-9bed-94d4cc22da44 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:54:15.372688 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.auth.core [None req-f1a1c9f0-7276-4833-9bed-94d4cc22da44 None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81578) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 01:54:15.373954 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-f1a1c9f0-7276-4833-9bed-94d4cc22da44 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:54:15.475240 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:54:15.477951 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 670/1348] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Aug 30 01:54:15 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 179 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 01:54:15.791417 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c641b280-ff2b-48d5-b3d3-7fbcc3a89574 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:54:15.792948 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-c641b280-ff2b-48d5-b3d3-7fbcc3a89574 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:54:15.904503 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c641b280-ff2b-48d5-b3d3-7fbcc3a89574 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:54:15.952705 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:54:15.985627 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5864b0b-5188-4cb9-b58b-baafd98439e6 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f3ea1580c56f4814a92bbb2acf542905', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4562f3951fbe465280073e60a11e44e2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:54:15.986245 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5864b0b-5188-4cb9-b58b-baafd98439e6 service neutron] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:54:15.986525 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5864b0b-5188-4cb9-b58b-baafd98439e6 service neutron] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:54:15.986796 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5864b0b-5188-4cb9-b58b-baafd98439e6 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:54:15.988414 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-e5864b0b-5188-4cb9-b58b-baafd98439e6 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:54:16.098431 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5864b0b-5188-4cb9-b58b-baafd98439e6 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:54:16.099856 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5864b0b-5188-4cb9-b58b-baafd98439e6 service neutron] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:54:16.117734 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 679/1349] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Aug 30 01:54:15 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 327 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 01:54:21.231513 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-caa4f3a1-34ae-45f9-813d-5bd4ae92af1d None None] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:54:21.231513 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-caa4f3a1-34ae-45f9-813d-5bd4ae92af1d None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:54:21.231513 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-caa4f3a1-34ae-45f9-813d-5bd4ae92af1d None None] PATH_INFO: `/` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:54:21.231513 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 671/1350] 10.222.0.27 () {54 vars in 921 bytes} [Tue Aug 30 01:54:21 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 01:54:21.235490 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ff43e2d-ac54-4dfd-9c55-dab4eb8faf36 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:54:21.235490 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ff43e2d-ac54-4dfd-9c55-dab4eb8faf36 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:54:21.235490 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ff43e2d-ac54-4dfd-9c55-dab4eb8faf36 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:54:21.329994 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-0ff43e2d-ac54-4dfd-9c55-dab4eb8faf36 None None] MFA Rules not processed for user `379536ff06fd4f719899e5dde732e63d`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 01:54:21.331309 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-0ff43e2d-ac54-4dfd-9c55-dab4eb8faf36 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:54:21.358461 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:54:21.519794 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 680/1351] 10.222.0.27 () {58 vars in 1018 bytes} [Tue Aug 30 01:54:21 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 287 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 30 01:54:25.773711 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0263e9c8-05e1-4fb7-84cd-80c45df57104 None None] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:54:25.774265 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0263e9c8-05e1-4fb7-84cd-80c45df57104 None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:54:25.774265 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0263e9c8-05e1-4fb7-84cd-80c45df57104 None None] PATH_INFO: `/` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:54:25.775867 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 672/1352] 10.222.0.27 () {54 vars in 921 bytes} [Tue Aug 30 01:54:25 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 01:54:25.783150 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c058cd1-af7d-44ac-9334-fe33b7aee106 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:54:25.783683 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c058cd1-af7d-44ac-9334-fe33b7aee106 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:54:25.783946 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c058cd1-af7d-44ac-9334-fe33b7aee106 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:54:25.864017 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-4c058cd1-af7d-44ac-9334-fe33b7aee106 None None] MFA Rules not processed for user `379536ff06fd4f719899e5dde732e63d`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 01:54:25.865239 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-4c058cd1-af7d-44ac-9334-fe33b7aee106 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:54:25.941191 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:54:25.977760 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 681/1353] 10.222.0.27 () {58 vars in 1018 bytes} [Tue Aug 30 01:54:25 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 195 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 01:54:27.306876 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcf3cae6-49e5-42af-9272-0a7eeeb4132f None None] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:54:27.307447 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcf3cae6-49e5-42af-9272-0a7eeeb4132f None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:54:27.307447 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcf3cae6-49e5-42af-9272-0a7eeeb4132f None None] PATH_INFO: `/` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:54:27.308911 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 673/1354] 10.222.0.27 () {54 vars in 921 bytes} [Tue Aug 30 01:54:27 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 01:54:27.314554 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4212575-f99f-4ce4-adea-51a8e0f57e89 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:54:27.314997 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4212575-f99f-4ce4-adea-51a8e0f57e89 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:54:27.315094 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4212575-f99f-4ce4-adea-51a8e0f57e89 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:54:27.400560 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-b4212575-f99f-4ce4-adea-51a8e0f57e89 None None] MFA Rules not processed for user `379536ff06fd4f719899e5dde732e63d`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 01:54:27.401801 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-b4212575-f99f-4ce4-adea-51a8e0f57e89 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:54:27.512270 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 682/1355] 10.222.0.27 () {58 vars in 1018 bytes} [Tue Aug 30 01:54:27 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 199 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 01:54:30.903435 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85b32a8b-8a65-4338-afef-a9d54b038286 None None] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:54:30.903435 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85b32a8b-8a65-4338-afef-a9d54b038286 None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:54:30.903435 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85b32a8b-8a65-4338-afef-a9d54b038286 None None] PATH_INFO: `/` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:54:30.904501 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 674/1356] 10.222.0.27 () {54 vars in 927 bytes} [Tue Aug 30 01:54:30 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 01:54:30.911132 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eedb284f-bc89-41b8-aa7c-26ed95c3b450 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:54:30.911586 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eedb284f-bc89-41b8-aa7c-26ed95c3b450 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:54:30.911773 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eedb284f-bc89-41b8-aa7c-26ed95c3b450 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:54:30.957694 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:54:31.004774 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-eedb284f-bc89-41b8-aa7c-26ed95c3b450 None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 01:54:31.006022 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-eedb284f-bc89-41b8-aa7c-26ed95c3b450 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:54:31.138416 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 683/1357] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Aug 30 01:54:30 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 228 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 01:54:31.451766 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94f00a55-ff00-432a-86e2-c07e14b6acad None None] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:54:31.452671 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94f00a55-ff00-432a-86e2-c07e14b6acad None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:54:31.453170 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94f00a55-ff00-432a-86e2-c07e14b6acad None None] PATH_INFO: `/` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:54:31.455059 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 675/1358] 10.222.0.27 () {54 vars in 962 bytes} [Tue Aug 30 01:54:31 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 01:54:31.467604 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2dc06cb-7306-44e9-b3b2-ecdc726aeeef None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:54:31.467922 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2dc06cb-7306-44e9-b3b2-ecdc726aeeef None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:54:31.468352 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2dc06cb-7306-44e9-b3b2-ecdc726aeeef None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:54:31.564351 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-a2dc06cb-7306-44e9-b3b2-ecdc726aeeef None None] MFA Rules not processed for user `d64f2e71361644589bec8a06e6eceed1`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 01:54:31.565606 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-a2dc06cb-7306-44e9-b3b2-ecdc726aeeef None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:54:31.630632 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:54:31.699911 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 684/1359] 10.222.0.27 () {58 vars in 1059 bytes} [Tue Aug 30 01:54:31 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 234 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 01:54:31.708154 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94f00a55-ff00-432a-86e2-c07e14b6acad None None] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:54:31.709682 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-94f00a55-ff00-432a-86e2-c07e14b6acad None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:54:31.869499 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94f00a55-ff00-432a-86e2-c07e14b6acad None None] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:54:31.948010 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:54:31.991359 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b09fd13-44bd-4e55-838c-205384c5e3e4 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd64f2e71361644589bec8a06e6eceed1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4562f3951fbe465280073e60a11e44e2', 'project_domain_id': 'default', 'roles': ['member', 'service', 'admin', 'ResellerAdmin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:54:31.991956 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b09fd13-44bd-4e55-838c-205384c5e3e4 service nova] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:54:31.992233 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b09fd13-44bd-4e55-838c-205384c5e3e4 service nova] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:54:31.992499 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b09fd13-44bd-4e55-838c-205384c5e3e4 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:54:31.994202 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-5b09fd13-44bd-4e55-838c-205384c5e3e4 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:54:32.086639 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b09fd13-44bd-4e55-838c-205384c5e3e4 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:54:32.088301 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b09fd13-44bd-4e55-838c-205384c5e3e4 service nova] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:54:32.107850 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 676/1360] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Aug 30 01:54:31 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 400 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 01:54:34.267427 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebe45146-e87d-48e1-acd0-8711b63ea8b4 None None] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:54:34.267968 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebe45146-e87d-48e1-acd0-8711b63ea8b4 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:54:34.267968 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebe45146-e87d-48e1-acd0-8711b63ea8b4 None None] PATH_INFO: `/` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:54:34.269443 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 685/1361] 10.222.0.27 () {54 vars in 927 bytes} [Tue Aug 30 01:54:34 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 01:54:34.274521 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-145b25bb-fa06-4f8d-932a-429187684507 None None] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:54:34.274945 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-145b25bb-fa06-4f8d-932a-429187684507 None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:54:34.275063 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-145b25bb-fa06-4f8d-932a-429187684507 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:54:34.348376 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.auth.core [None req-145b25bb-fa06-4f8d-932a-429187684507 None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81578) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 01:54:34.349792 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-145b25bb-fa06-4f8d-932a-429187684507 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:54:34.389371 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:54:34.463068 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 677/1362] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Aug 30 01:54:34 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 190 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 01:54:34.762532 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebe45146-e87d-48e1-acd0-8711b63ea8b4 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:54:34.764055 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-ebe45146-e87d-48e1-acd0-8711b63ea8b4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:54:34.928818 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebe45146-e87d-48e1-acd0-8711b63ea8b4 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:54:34.938941 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:54:35.053586 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-172111f8-8d87-4366-8dcb-61ab32ebcb7d service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd64f2e71361644589bec8a06e6eceed1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4562f3951fbe465280073e60a11e44e2', 'project_domain_id': 'default', 'roles': ['member', 'service', 'admin', 'ResellerAdmin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:54:35.054245 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-172111f8-8d87-4366-8dcb-61ab32ebcb7d service nova] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:54:35.054439 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-172111f8-8d87-4366-8dcb-61ab32ebcb7d service nova] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:54:35.054890 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-172111f8-8d87-4366-8dcb-61ab32ebcb7d service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:54:35.056478 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-172111f8-8d87-4366-8dcb-61ab32ebcb7d service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:54:35.153855 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-172111f8-8d87-4366-8dcb-61ab32ebcb7d service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:54:35.156048 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-172111f8-8d87-4366-8dcb-61ab32ebcb7d service nova] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:54:35.175371 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 686/1363] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Aug 30 01:54:34 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 413 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 01:54:37.509297 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-129dc579-2761-4ad8-944d-d141c5c07ced None None] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:54:37.509937 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-129dc579-2761-4ad8-944d-d141c5c07ced None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:54:37.509937 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-129dc579-2761-4ad8-944d-d141c5c07ced None None] PATH_INFO: `/` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:54:37.511496 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 678/1364] 10.222.0.27 () {54 vars in 927 bytes} [Tue Aug 30 01:54:37 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 01:54:37.516837 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d23baf0-6905-4fae-a7ff-425fa1366e19 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:54:37.517261 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d23baf0-6905-4fae-a7ff-425fa1366e19 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:54:37.517470 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d23baf0-6905-4fae-a7ff-425fa1366e19 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:54:37.534435 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:54:37.589415 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-3d23baf0-6905-4fae-a7ff-425fa1366e19 None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 01:54:37.590664 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-3d23baf0-6905-4fae-a7ff-425fa1366e19 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:54:37.695904 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 687/1365] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Aug 30 01:54:37 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 178 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 01:54:37.993268 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87be573e-87cb-40c1-94ff-e707ad724aa1 None None] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:54:37.993607 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87be573e-87cb-40c1-94ff-e707ad724aa1 None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:54:37.994523 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87be573e-87cb-40c1-94ff-e707ad724aa1 None None] PATH_INFO: `/` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:54:37.996360 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 679/1366] 10.222.0.27 () {54 vars in 962 bytes} [Tue Aug 30 01:54:37 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 01:54:38.002438 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3f04557-5fc0-4061-bc8f-26f85cf76f92 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:54:38.002785 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3f04557-5fc0-4061-bc8f-26f85cf76f92 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:54:38.004941 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3f04557-5fc0-4061-bc8f-26f85cf76f92 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:54:38.083560 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-d3f04557-5fc0-4061-bc8f-26f85cf76f92 None None] MFA Rules not processed for user `d64f2e71361644589bec8a06e6eceed1`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 01:54:38.084773 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-d3f04557-5fc0-4061-bc8f-26f85cf76f92 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:54:38.123470 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:54:38.224730 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 688/1367] 10.222.0.27 () {58 vars in 1059 bytes} [Tue Aug 30 01:54:38 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 223 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 01:54:38.234123 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87be573e-87cb-40c1-94ff-e707ad724aa1 None None] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:54:38.236118 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-87be573e-87cb-40c1-94ff-e707ad724aa1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:54:38.406766 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87be573e-87cb-40c1-94ff-e707ad724aa1 None None] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:54:38.428184 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:54:38.530612 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96a36969-d440-4192-bb8d-2cfb3a6fa307 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd64f2e71361644589bec8a06e6eceed1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4562f3951fbe465280073e60a11e44e2', 'project_domain_id': 'default', 'roles': ['member', 'service', 'admin', 'ResellerAdmin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:54:38.530612 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96a36969-d440-4192-bb8d-2cfb3a6fa307 service nova] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:54:38.532939 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96a36969-d440-4192-bb8d-2cfb3a6fa307 service nova] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:54:38.533364 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96a36969-d440-4192-bb8d-2cfb3a6fa307 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:54:38.535923 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-96a36969-d440-4192-bb8d-2cfb3a6fa307 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:54:38.665449 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96a36969-d440-4192-bb8d-2cfb3a6fa307 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:54:38.667090 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96a36969-d440-4192-bb8d-2cfb3a6fa307 service nova] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:54:38.685551 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 680/1368] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Aug 30 01:54:38 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 452 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 01:54:39.200664 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d3bf220-8ec3-4eb6-8093-8a36d2d3db9b None None] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:54:39.201175 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d3bf220-8ec3-4eb6-8093-8a36d2d3db9b None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:54:39.201175 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d3bf220-8ec3-4eb6-8093-8a36d2d3db9b None None] PATH_INFO: `/` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:54:39.203281 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 689/1369] 10.222.0.27 () {54 vars in 927 bytes} [Tue Aug 30 01:54:39 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 01:54:39.209539 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bb9af09-4968-4e7b-8b24-ccc6f98d85fa None None] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:54:39.209925 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bb9af09-4968-4e7b-8b24-ccc6f98d85fa None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:54:39.210259 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bb9af09-4968-4e7b-8b24-ccc6f98d85fa None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:54:39.245120 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:54:39.282512 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.auth.core [None req-6bb9af09-4968-4e7b-8b24-ccc6f98d85fa None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81578) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 01:54:39.283752 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-6bb9af09-4968-4e7b-8b24-ccc6f98d85fa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:54:39.393901 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 681/1370] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Aug 30 01:54:39 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 185 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 01:54:39.981144 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abfedcfd-787f-45d2-93f1-a400595fb4ff None None] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:54:39.981144 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abfedcfd-787f-45d2-93f1-a400595fb4ff None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:54:39.981144 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abfedcfd-787f-45d2-93f1-a400595fb4ff None None] PATH_INFO: `/` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:54:39.984857 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 690/1371] 10.222.0.27 () {54 vars in 964 bytes} [Tue Aug 30 01:54:39 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 01:54:39.989361 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf28dd63-ed7e-4585-93f3-5a8b367b3933 None None] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:54:39.989880 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf28dd63-ed7e-4585-93f3-5a8b367b3933 None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:54:39.990110 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf28dd63-ed7e-4585-93f3-5a8b367b3933 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:54:40.072258 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.auth.core [None req-bf28dd63-ed7e-4585-93f3-5a8b367b3933 None None] MFA Rules not processed for user `ab936b3767a8417aa80c8c8f1b84b667`. Rule list: `[]` (Enabled: `True`). {{(pid=81578) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 01:54:40.073585 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-bf28dd63-ed7e-4585-93f3-5a8b367b3933 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:54:40.122815 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:54:40.153950 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 682/1372] 10.222.0.27 () {58 vars in 1061 bytes} [Tue Aug 30 01:54:39 2022] POST /identity/v3/auth/tokens => generated 3120 bytes in 165 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 01:54:40.161735 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abfedcfd-787f-45d2-93f1-a400595fb4ff None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:54:40.163353 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-abfedcfd-787f-45d2-93f1-a400595fb4ff None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:54:40.275591 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abfedcfd-787f-45d2-93f1-a400595fb4ff None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:54:40.320010 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:54:40.356464 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01e10c89-4394-427f-a169-79e19b63faf0 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ab936b3767a8417aa80c8c8f1b84b667', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4562f3951fbe465280073e60a11e44e2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:54:40.357079 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01e10c89-4394-427f-a169-79e19b63faf0 service cinder] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:54:40.357357 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01e10c89-4394-427f-a169-79e19b63faf0 service cinder] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:54:40.357662 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01e10c89-4394-427f-a169-79e19b63faf0 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:54:40.360175 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-01e10c89-4394-427f-a169-79e19b63faf0 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:54:40.486613 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01e10c89-4394-427f-a169-79e19b63faf0 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:54:40.488596 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01e10c89-4394-427f-a169-79e19b63faf0 service cinder] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:54:40.513758 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 691/1373] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Aug 30 01:54:40 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 352 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 01:54:40.834209 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cc30161-1cd7-4be3-acec-1f9cc3516693 None None] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:54:40.834494 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cc30161-1cd7-4be3-acec-1f9cc3516693 None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:54:40.834765 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cc30161-1cd7-4be3-acec-1f9cc3516693 None None] PATH_INFO: `/` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:54:40.836097 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 683/1374] 10.222.0.27 () {54 vars in 927 bytes} [Tue Aug 30 01:54:40 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 01:54:40.841205 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c10f592-41d2-47cd-a2ec-884275d6d0d8 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:54:40.841516 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c10f592-41d2-47cd-a2ec-884275d6d0d8 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:54:40.841792 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c10f592-41d2-47cd-a2ec-884275d6d0d8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:54:40.908270 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-6c10f592-41d2-47cd-a2ec-884275d6d0d8 None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 01:54:40.909486 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-6c10f592-41d2-47cd-a2ec-884275d6d0d8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:54:40.930912 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:54:41.007192 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 692/1375] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Aug 30 01:54:40 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 167 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 01:54:41.293033 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cc30161-1cd7-4be3-acec-1f9cc3516693 None None] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:54:41.294651 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-6cc30161-1cd7-4be3-acec-1f9cc3516693 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:54:41.447591 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cc30161-1cd7-4be3-acec-1f9cc3516693 None None] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:54:41.499684 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:54:41.562619 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2dc4fded-8657-4b1d-b6de-411b5935fb6a service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd64f2e71361644589bec8a06e6eceed1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4562f3951fbe465280073e60a11e44e2', 'project_domain_id': 'default', 'roles': ['member', 'service', 'admin', 'ResellerAdmin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:54:41.563223 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dc4fded-8657-4b1d-b6de-411b5935fb6a service nova] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:54:41.563500 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dc4fded-8657-4b1d-b6de-411b5935fb6a service nova] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:54:41.563766 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dc4fded-8657-4b1d-b6de-411b5935fb6a service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:54:41.565425 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-2dc4fded-8657-4b1d-b6de-411b5935fb6a service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:54:41.654038 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2dc4fded-8657-4b1d-b6de-411b5935fb6a service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:54:41.656176 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2dc4fded-8657-4b1d-b6de-411b5935fb6a service nova] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:54:41.674631 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 684/1376] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Aug 30 01:54:41 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 382 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 01:54:43.743333 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d70ea95d-f2a2-40a0-a4a8-75030e7b5883 None None] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:54:43.743886 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d70ea95d-f2a2-40a0-a4a8-75030e7b5883 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:54:43.743886 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d70ea95d-f2a2-40a0-a4a8-75030e7b5883 None None] PATH_INFO: `/` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:54:43.745237 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 693/1377] 10.222.0.27 () {54 vars in 927 bytes} [Tue Aug 30 01:54:43 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 01:54:43.750830 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3172ec8-8e5a-43c2-8ec9-0fc22d6394af None None] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:54:43.751250 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3172ec8-8e5a-43c2-8ec9-0fc22d6394af None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:54:43.751345 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3172ec8-8e5a-43c2-8ec9-0fc22d6394af None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:54:43.832393 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.auth.core [None req-f3172ec8-8e5a-43c2-8ec9-0fc22d6394af None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81578) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 01:54:43.833695 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-f3172ec8-8e5a-43c2-8ec9-0fc22d6394af None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:54:43.840299 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:54:43.930661 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 685/1378] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Aug 30 01:54:43 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 181 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 01:54:44.230301 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d70ea95d-f2a2-40a0-a4a8-75030e7b5883 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:54:44.232394 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-d70ea95d-f2a2-40a0-a4a8-75030e7b5883 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:54:44.417651 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d70ea95d-f2a2-40a0-a4a8-75030e7b5883 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:54:44.422718 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:54:44.547432 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97c6e324-7548-4bb0-97f1-b87f374863a4 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd64f2e71361644589bec8a06e6eceed1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4562f3951fbe465280073e60a11e44e2', 'project_domain_id': 'default', 'roles': ['member', 'service', 'admin', 'ResellerAdmin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:54:44.548041 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97c6e324-7548-4bb0-97f1-b87f374863a4 service nova] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:54:44.548319 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97c6e324-7548-4bb0-97f1-b87f374863a4 service nova] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:54:44.548593 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97c6e324-7548-4bb0-97f1-b87f374863a4 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:54:44.553230 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-97c6e324-7548-4bb0-97f1-b87f374863a4 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:54:44.644424 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97c6e324-7548-4bb0-97f1-b87f374863a4 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:54:44.646157 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97c6e324-7548-4bb0-97f1-b87f374863a4 service nova] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:54:44.665583 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 694/1379] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Aug 30 01:54:44 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 436 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 01:54:46.836506 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-303f9dd0-ecc0-426f-bc3b-8ae48bc2a90b None None] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:54:46.837033 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-303f9dd0-ecc0-426f-bc3b-8ae48bc2a90b None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:54:46.837262 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-303f9dd0-ecc0-426f-bc3b-8ae48bc2a90b None None] PATH_INFO: `/` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:54:46.838872 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 686/1380] 10.222.0.27 () {54 vars in 927 bytes} [Tue Aug 30 01:54:46 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 01:54:46.845185 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91a45f30-bc67-4ec3-ac59-742c90c6ce14 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:54:46.846246 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91a45f30-bc67-4ec3-ac59-742c90c6ce14 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:54:46.846246 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91a45f30-bc67-4ec3-ac59-742c90c6ce14 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:54:46.858812 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:54:46.922441 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-91a45f30-bc67-4ec3-ac59-742c90c6ce14 None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 01:54:46.923651 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-91a45f30-bc67-4ec3-ac59-742c90c6ce14 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:54:47.054454 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 695/1381] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Aug 30 01:54:46 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 210 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 01:54:47.376708 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-303f9dd0-ecc0-426f-bc3b-8ae48bc2a90b None None] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:54:47.378385 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-303f9dd0-ecc0-426f-bc3b-8ae48bc2a90b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:54:47.554807 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:54:47.566086 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-303f9dd0-ecc0-426f-bc3b-8ae48bc2a90b None None] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:54:47.688837 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0e3aa20-4ab9-473a-b3c7-74c0a92767e7 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd64f2e71361644589bec8a06e6eceed1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4562f3951fbe465280073e60a11e44e2', 'project_domain_id': 'default', 'roles': ['member', 'service', 'admin', 'ResellerAdmin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:54:47.689311 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0e3aa20-4ab9-473a-b3c7-74c0a92767e7 service nova] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:54:47.689614 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0e3aa20-4ab9-473a-b3c7-74c0a92767e7 service nova] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:54:47.689884 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0e3aa20-4ab9-473a-b3c7-74c0a92767e7 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:54:47.691496 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-c0e3aa20-4ab9-473a-b3c7-74c0a92767e7 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:54:47.800181 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0e3aa20-4ab9-473a-b3c7-74c0a92767e7 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:54:47.801932 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0e3aa20-4ab9-473a-b3c7-74c0a92767e7 service nova] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:54:47.814035 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:54:47.822830 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 687/1382] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Aug 30 01:54:47 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 447 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 01:54:50.059327 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ee4c909-90a3-4e67-8275-58d1df81a735 None None] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:54:50.060057 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ee4c909-90a3-4e67-8275-58d1df81a735 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:54:50.060057 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ee4c909-90a3-4e67-8275-58d1df81a735 None None] PATH_INFO: `/` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:54:50.061861 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 696/1383] 10.222.0.27 () {54 vars in 927 bytes} [Tue Aug 30 01:54:50 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 01:54:50.067192 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6357a7bc-cfee-42d8-9309-c03b520c54ce None None] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:54:50.067706 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6357a7bc-cfee-42d8-9309-c03b520c54ce None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:54:50.067868 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6357a7bc-cfee-42d8-9309-c03b520c54ce None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:54:50.137192 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.auth.core [None req-6357a7bc-cfee-42d8-9309-c03b520c54ce None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81578) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 01:54:50.138618 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-6357a7bc-cfee-42d8-9309-c03b520c54ce None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:54:50.243980 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 688/1384] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Aug 30 01:54:50 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 178 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 01:54:50.559718 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ee4c909-90a3-4e67-8275-58d1df81a735 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:54:50.562479 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-8ee4c909-90a3-4e67-8275-58d1df81a735 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:54:50.668328 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:54:50.738580 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ee4c909-90a3-4e67-8275-58d1df81a735 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:54:50.879354 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-189f60bc-5a58-4d5a-8b91-dbbdeeaf8b2c service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd64f2e71361644589bec8a06e6eceed1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4562f3951fbe465280073e60a11e44e2', 'project_domain_id': 'default', 'roles': ['member', 'service', 'admin', 'ResellerAdmin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:54:50.880085 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-189f60bc-5a58-4d5a-8b91-dbbdeeaf8b2c service nova] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:54:50.880448 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-189f60bc-5a58-4d5a-8b91-dbbdeeaf8b2c service nova] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:54:50.880789 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-189f60bc-5a58-4d5a-8b91-dbbdeeaf8b2c service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:54:50.882834 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-189f60bc-5a58-4d5a-8b91-dbbdeeaf8b2c service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:54:50.943952 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:54:50.986227 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-189f60bc-5a58-4d5a-8b91-dbbdeeaf8b2c service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:54:50.988447 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-189f60bc-5a58-4d5a-8b91-dbbdeeaf8b2c service nova] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:54:51.012209 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 697/1385] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Aug 30 01:54:50 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 453 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 01:54:53.072575 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-660bfe7a-c0e8-44fc-b4f3-0f1e0c158d58 None None] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:54:53.073284 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-660bfe7a-c0e8-44fc-b4f3-0f1e0c158d58 None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:54:53.073398 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-660bfe7a-c0e8-44fc-b4f3-0f1e0c158d58 None None] PATH_INFO: `/` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:54:53.074925 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 689/1386] 10.222.0.27 () {54 vars in 927 bytes} [Tue Aug 30 01:54:53 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 01:54:53.081016 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba89ac65-1bb7-4db8-89da-b55c78be443f None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:54:53.081549 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba89ac65-1bb7-4db8-89da-b55c78be443f None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:54:53.081756 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba89ac65-1bb7-4db8-89da-b55c78be443f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:54:53.156811 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-ba89ac65-1bb7-4db8-89da-b55c78be443f None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 01:54:53.158134 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-ba89ac65-1bb7-4db8-89da-b55c78be443f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:54:53.259663 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 698/1387] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Aug 30 01:54:53 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 180 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 01:54:53.569523 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-660bfe7a-c0e8-44fc-b4f3-0f1e0c158d58 None None] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:54:53.571054 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-660bfe7a-c0e8-44fc-b4f3-0f1e0c158d58 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:54:53.650298 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:54:53.721254 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-660bfe7a-c0e8-44fc-b4f3-0f1e0c158d58 None None] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:54:53.849272 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-415854db-a09c-4393-b7a7-f7696da9b372 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd64f2e71361644589bec8a06e6eceed1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4562f3951fbe465280073e60a11e44e2', 'project_domain_id': 'default', 'roles': ['member', 'service', 'admin', 'ResellerAdmin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:54:53.849854 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-415854db-a09c-4393-b7a7-f7696da9b372 service nova] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:54:53.850150 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-415854db-a09c-4393-b7a7-f7696da9b372 service nova] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:54:53.850430 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-415854db-a09c-4393-b7a7-f7696da9b372 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:54:53.852158 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-415854db-a09c-4393-b7a7-f7696da9b372 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:54:53.893588 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:54:53.948666 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-415854db-a09c-4393-b7a7-f7696da9b372 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:54:53.950754 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-415854db-a09c-4393-b7a7-f7696da9b372 service nova] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:54:53.973854 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 690/1388] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Aug 30 01:54:53 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 405 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 01:54:56.061114 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a5d410b-6508-4abd-ad9a-f35657432e85 None None] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:54:56.061715 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a5d410b-6508-4abd-ad9a-f35657432e85 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:54:56.061715 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a5d410b-6508-4abd-ad9a-f35657432e85 None None] PATH_INFO: `/` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:54:56.063079 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 699/1389] 10.222.0.27 () {54 vars in 927 bytes} [Tue Aug 30 01:54:56 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 01:54:56.068324 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd228688-ce1a-4cbb-9ed0-8bc3ee74fbdf None None] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:54:56.068754 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd228688-ce1a-4cbb-9ed0-8bc3ee74fbdf None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:54:56.068872 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd228688-ce1a-4cbb-9ed0-8bc3ee74fbdf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:54:56.084440 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb71ad8a-a8c0-4eb9-9d28-4fd1a7d04ebe None None] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:54:56.084734 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb71ad8a-a8c0-4eb9-9d28-4fd1a7d04ebe None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:54:56.085001 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb71ad8a-a8c0-4eb9-9d28-4fd1a7d04ebe None None] PATH_INFO: `/` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:54:56.086426 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 700/1390] 10.222.0.27 () {54 vars in 927 bytes} [Tue Aug 30 01:54:56 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 30 01:54:56.091477 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-144d80c6-3525-4b53-8cd8-93101132e3ec None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:54:56.092308 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-144d80c6-3525-4b53-8cd8-93101132e3ec None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:54:56.092308 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-144d80c6-3525-4b53-8cd8-93101132e3ec None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:54:56.128268 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:54:56.139112 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.auth.core [None req-bd228688-ce1a-4cbb-9ed0-8bc3ee74fbdf None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81578) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 01:54:56.147628 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-bd228688-ce1a-4cbb-9ed0-8bc3ee74fbdf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:54:56.160090 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-144d80c6-3525-4b53-8cd8-93101132e3ec None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 01:54:56.161294 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-144d80c6-3525-4b53-8cd8-93101132e3ec None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:54:56.239677 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 691/1391] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Aug 30 01:54:56 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 172 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 01:54:56.259099 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 701/1392] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Aug 30 01:54:56 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 168 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 30 01:54:56.536812 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd228688-ce1a-4cbb-9ed0-8bc3ee74fbdf None None] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:54:56.538038 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-bd228688-ce1a-4cbb-9ed0-8bc3ee74fbdf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:54:56.553757 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:54:56.566155 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09e0b7dc-a602-40f9-9527-525e7473add7 None None] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:54:56.566441 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09e0b7dc-a602-40f9-9527-525e7473add7 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:54:56.566710 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09e0b7dc-a602-40f9-9527-525e7473add7 None None] PATH_INFO: `/` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:54:56.568222 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 702/1393] 10.222.0.27 () {54 vars in 967 bytes} [Tue Aug 30 01:54:56 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 30 01:54:56.573220 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b77ba70-08b4-45e8-9d01-60285cbe97c9 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:54:56.573530 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b77ba70-08b4-45e8-9d01-60285cbe97c9 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:54:56.573809 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b77ba70-08b4-45e8-9d01-60285cbe97c9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:54:56.657408 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd228688-ce1a-4cbb-9ed0-8bc3ee74fbdf None None] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:54:56.660899 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-2b77ba70-08b4-45e8-9d01-60285cbe97c9 None None] MFA Rules not processed for user `3caff62093d748f48596574836e12e42`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 01:54:56.662148 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-2b77ba70-08b4-45e8-9d01-60285cbe97c9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:54:56.717563 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:54:56.746925 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 703/1394] 10.222.0.27 () {58 vars in 1064 bytes} [Tue Aug 30 01:54:56 2022] POST /identity/v3/auth/tokens => generated 3120 bytes in 174 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 30 01:54:56.754691 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b77ba70-08b4-45e8-9d01-60285cbe97c9 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:54:56.762126 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b77ba70-08b4-45e8-9d01-60285cbe97c9 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:54:56.783223 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5809b92-43d2-44df-b009-f782a3b1ef5a service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd64f2e71361644589bec8a06e6eceed1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4562f3951fbe465280073e60a11e44e2', 'project_domain_id': 'default', 'roles': ['member', 'service', 'admin', 'ResellerAdmin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:54:56.783767 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5809b92-43d2-44df-b009-f782a3b1ef5a service nova] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:54:56.784049 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5809b92-43d2-44df-b009-f782a3b1ef5a service nova] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:54:56.784318 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5809b92-43d2-44df-b009-f782a3b1ef5a service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:54:56.785947 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-d5809b92-43d2-44df-b009-f782a3b1ef5a service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:54:56.795409 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:54:56.837511 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-210630f4-0f35-43e8-bcdb-b1106b6510df service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3caff62093d748f48596574836e12e42', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4562f3951fbe465280073e60a11e44e2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:54:56.838124 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-210630f4-0f35-43e8-bcdb-b1106b6510df service glance] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:54:56.838403 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-210630f4-0f35-43e8-bcdb-b1106b6510df service glance] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:54:56.838673 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-210630f4-0f35-43e8-bcdb-b1106b6510df service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:54:56.840244 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-210630f4-0f35-43e8-bcdb-b1106b6510df service glance] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:54:56.879538 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5809b92-43d2-44df-b009-f782a3b1ef5a service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:54:56.881137 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5809b92-43d2-44df-b009-f782a3b1ef5a service nova] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:54:56.899643 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 692/1395] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Aug 30 01:54:56 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 363 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 01:54:56.955737 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-210630f4-0f35-43e8-bcdb-b1106b6510df service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:54:56.957176 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-210630f4-0f35-43e8-bcdb-b1106b6510df service glance] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:54:56.972179 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:54:56.976949 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 704/1396] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Aug 30 01:54:56 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 223 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 30 01:54:57.827039 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92841e2d-6da3-42a5-b87a-d1804d743874 None None] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:54:57.827543 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92841e2d-6da3-42a5-b87a-d1804d743874 None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:54:57.827647 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92841e2d-6da3-42a5-b87a-d1804d743874 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:54:57.899253 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.auth.core [None req-92841e2d-6da3-42a5-b87a-d1804d743874 None None] MFA Rules not processed for user `121c374d49a64335ab74a431cfbc1320`. Rule list: `[]` (Enabled: `True`). {{(pid=81578) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 01:54:57.900472 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-92841e2d-6da3-42a5-b87a-d1804d743874 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:54:57.985540 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:54:57.988869 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 693/1397] 10.222.0.27 () {58 vars in 1009 bytes} [Tue Aug 30 01:54:57 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 164 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 01:54:58.033376 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06629736-9473-4cde-9e4d-23adad7175af None None] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:54:58.033908 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06629736-9473-4cde-9e4d-23adad7175af None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:54:58.034025 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06629736-9473-4cde-9e4d-23adad7175af None None] PATH_INFO: `/` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:54:58.035280 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 705/1398] 10.222.0.27 () {54 vars in 943 bytes} [Tue Aug 30 01:54:58 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 01:54:58.041835 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3558af7-bca5-4714-ba16-c4c8ec4ea4a2 None None] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:54:58.042097 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3558af7-bca5-4714-ba16-c4c8ec4ea4a2 None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:54:58.042370 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3558af7-bca5-4714-ba16-c4c8ec4ea4a2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:54:58.120055 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.auth.core [None req-e3558af7-bca5-4714-ba16-c4c8ec4ea4a2 None None] MFA Rules not processed for user `b4f0008fde424f22bc1f7307cf422774`. Rule list: `[]` (Enabled: `True`). {{(pid=81578) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 01:54:58.121264 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-e3558af7-bca5-4714-ba16-c4c8ec4ea4a2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:54:58.230637 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 694/1399] 10.222.0.27 () {58 vars in 1040 bytes} [Tue Aug 30 01:54:58 2022] POST /identity/v3/auth/tokens => generated 3304 bytes in 190 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 01:54:58.240768 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06629736-9473-4cde-9e4d-23adad7175af None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:54:58.242313 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-06629736-9473-4cde-9e4d-23adad7175af None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:54:58.383910 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06629736-9473-4cde-9e4d-23adad7175af None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:54:58.473023 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:54:58.493325 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-563efa4d-48f4-441c-8b29-bdba52e0fb2c service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b4f0008fde424f22bc1f7307cf422774', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4562f3951fbe465280073e60a11e44e2', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:54:58.493905 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-563efa4d-48f4-441c-8b29-bdba52e0fb2c service swift] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:54:58.494324 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-563efa4d-48f4-441c-8b29-bdba52e0fb2c service swift] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:54:58.494593 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-563efa4d-48f4-441c-8b29-bdba52e0fb2c service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:54:58.496195 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-563efa4d-48f4-441c-8b29-bdba52e0fb2c service swift] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:54:58.586022 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-563efa4d-48f4-441c-8b29-bdba52e0fb2c service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:54:58.587702 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-563efa4d-48f4-441c-8b29-bdba52e0fb2c service swift] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:54:58.595624 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 706/1400] 10.222.0.27 () {60 vars in 1384 bytes} [Tue Aug 30 01:54:58 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 355 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Aug 30 01:54:59.009447 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd9452d7-f522-43d9-8620-606d29ca8cba None None] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:54:59.009903 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd9452d7-f522-43d9-8620-606d29ca8cba None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:54:59.010147 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd9452d7-f522-43d9-8620-606d29ca8cba None None] PATH_INFO: `/` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:54:59.011491 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 695/1401] 10.222.0.27 () {54 vars in 927 bytes} [Tue Aug 30 01:54:59 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 01:54:59.023453 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53b6ec17-7d15-4dc2-a774-c231a41fa054 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:54:59.023635 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53b6ec17-7d15-4dc2-a774-c231a41fa054 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:54:59.023905 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53b6ec17-7d15-4dc2-a774-c231a41fa054 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:54:59.107896 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-53b6ec17-7d15-4dc2-a774-c231a41fa054 None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 01:54:59.109204 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-53b6ec17-7d15-4dc2-a774-c231a41fa054 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:54:59.185809 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:54:59.221489 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 707/1402] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Aug 30 01:54:59 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 198 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 01:54:59.521352 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd9452d7-f522-43d9-8620-606d29ca8cba None None] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:54:59.522980 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-dd9452d7-f522-43d9-8620-606d29ca8cba None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:54:59.597858 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:54:59.696439 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd9452d7-f522-43d9-8620-606d29ca8cba None None] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:54:59.830573 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df04a7e4-4134-491e-9abf-c27f7dcde0de service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd64f2e71361644589bec8a06e6eceed1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4562f3951fbe465280073e60a11e44e2', 'project_domain_id': 'default', 'roles': ['member', 'service', 'admin', 'ResellerAdmin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:54:59.831579 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df04a7e4-4134-491e-9abf-c27f7dcde0de service nova] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:54:59.831947 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df04a7e4-4134-491e-9abf-c27f7dcde0de service nova] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:54:59.832221 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df04a7e4-4134-491e-9abf-c27f7dcde0de service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:54:59.833864 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-df04a7e4-4134-491e-9abf-c27f7dcde0de service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:54:59.855164 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:54:59.943288 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df04a7e4-4134-491e-9abf-c27f7dcde0de service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:54:59.945295 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df04a7e4-4134-491e-9abf-c27f7dcde0de service nova] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:54:59.968416 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 696/1403] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Aug 30 01:54:59 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 448 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 01:55:02.722564 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d00b3004-ea18-47ed-a014-c8c1a858bfa2 None None] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:55:02.723146 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d00b3004-ea18-47ed-a014-c8c1a858bfa2 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:55:02.725644 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d00b3004-ea18-47ed-a014-c8c1a858bfa2 None None] PATH_INFO: `/` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:55:02.727205 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 708/1404] 10.222.0.27 () {54 vars in 927 bytes} [Tue Aug 30 01:55:02 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 01:55:02.732663 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b0bbec1-4424-4b50-945b-003be7aefe3e None None] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:55:02.733131 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b0bbec1-4424-4b50-945b-003be7aefe3e None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:55:02.733238 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b0bbec1-4424-4b50-945b-003be7aefe3e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:55:02.805148 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.auth.core [None req-1b0bbec1-4424-4b50-945b-003be7aefe3e None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81578) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 01:55:02.806413 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-1b0bbec1-4424-4b50-945b-003be7aefe3e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:55:02.923430 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:55:02.930739 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 697/1405] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Aug 30 01:55:02 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 200 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 01:55:03.228066 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d00b3004-ea18-47ed-a014-c8c1a858bfa2 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:55:03.229647 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-d00b3004-ea18-47ed-a014-c8c1a858bfa2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:55:03.405585 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d00b3004-ea18-47ed-a014-c8c1a858bfa2 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:55:03.468503 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:55:03.543037 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f0ce577-05b3-4f90-a3ce-99f1b60e964f service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd64f2e71361644589bec8a06e6eceed1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4562f3951fbe465280073e60a11e44e2', 'project_domain_id': 'default', 'roles': ['member', 'service', 'admin', 'ResellerAdmin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:55:03.543037 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f0ce577-05b3-4f90-a3ce-99f1b60e964f service nova] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:55:03.543373 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f0ce577-05b3-4f90-a3ce-99f1b60e964f service nova] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:55:03.543641 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f0ce577-05b3-4f90-a3ce-99f1b60e964f service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:55:03.545900 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-0f0ce577-05b3-4f90-a3ce-99f1b60e964f service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:55:03.646359 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f0ce577-05b3-4f90-a3ce-99f1b60e964f service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:55:03.647714 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f0ce577-05b3-4f90-a3ce-99f1b60e964f service nova] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:55:03.666677 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 709/1406] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Aug 30 01:55:03 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 439 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 01:55:05.792977 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd58af56-038a-4012-aa5b-442c79fb4376 None None] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:55:05.793549 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd58af56-038a-4012-aa5b-442c79fb4376 None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:55:05.793549 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd58af56-038a-4012-aa5b-442c79fb4376 None None] PATH_INFO: `/` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:55:05.795289 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 698/1407] 10.222.0.27 () {54 vars in 927 bytes} [Tue Aug 30 01:55:05 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 01:55:05.800645 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2137d4c5-0198-4557-b079-bf75ed8b2153 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:55:05.801142 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2137d4c5-0198-4557-b079-bf75ed8b2153 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:55:05.801249 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2137d4c5-0198-4557-b079-bf75ed8b2153 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:55:05.890656 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-2137d4c5-0198-4557-b079-bf75ed8b2153 None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 01:55:05.891914 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-2137d4c5-0198-4557-b079-bf75ed8b2153 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:55:05.905929 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:55:06.008768 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 710/1408] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Aug 30 01:55:05 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 209 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 01:55:06.305577 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd58af56-038a-4012-aa5b-442c79fb4376 None None] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:55:06.307501 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-fd58af56-038a-4012-aa5b-442c79fb4376 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:55:06.477554 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd58af56-038a-4012-aa5b-442c79fb4376 None None] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:55:06.555386 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:55:06.611146 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c99b96a-1e01-4c4f-b819-a8b7c0c6b3f9 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd64f2e71361644589bec8a06e6eceed1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4562f3951fbe465280073e60a11e44e2', 'project_domain_id': 'default', 'roles': ['member', 'service', 'admin', 'ResellerAdmin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:55:06.611789 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c99b96a-1e01-4c4f-b819-a8b7c0c6b3f9 service nova] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:55:06.612070 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c99b96a-1e01-4c4f-b819-a8b7c0c6b3f9 service nova] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:55:06.612343 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c99b96a-1e01-4c4f-b819-a8b7c0c6b3f9 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:55:06.613989 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-1c99b96a-1e01-4c4f-b819-a8b7c0c6b3f9 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:55:06.726919 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c99b96a-1e01-4c4f-b819-a8b7c0c6b3f9 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:55:06.728777 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c99b96a-1e01-4c4f-b819-a8b7c0c6b3f9 service nova] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:55:06.748580 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 699/1409] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Aug 30 01:55:06 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 443 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 01:55:16.223137 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=38 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:55:16.745792 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=64 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:55:22.444170 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-993efc02-333a-40ed-8728-872b0e35fe15 None None] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:55:22.444751 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-993efc02-333a-40ed-8728-872b0e35fe15 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:55:22.444751 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-993efc02-333a-40ed-8728-872b0e35fe15 None None] PATH_INFO: `/` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:55:22.446084 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 711/1410] 10.222.0.27 () {54 vars in 918 bytes} [Tue Aug 30 01:55:22 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 01:55:22.451834 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b21a6c4-9ee9-460d-8b1e-84449d76c05d None None] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:55:22.452282 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b21a6c4-9ee9-460d-8b1e-84449d76c05d None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:55:22.452395 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b21a6c4-9ee9-460d-8b1e-84449d76c05d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:55:22.526393 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.auth.core [None req-6b21a6c4-9ee9-460d-8b1e-84449d76c05d None None] MFA Rules not processed for user `379536ff06fd4f719899e5dde732e63d`. Rule list: `[]` (Enabled: `True`). {{(pid=81578) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 01:55:22.528325 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-6b21a6c4-9ee9-460d-8b1e-84449d76c05d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:55:22.633443 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 700/1411] 10.222.0.27 () {58 vars in 1015 bytes} [Tue Aug 30 01:55:22 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 183 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 01:55:22.648154 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cde60d4-aace-46c9-9df8-252cb4493f3b None None] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:55:22.648444 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cde60d4-aace-46c9-9df8-252cb4493f3b None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:55:22.648725 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cde60d4-aace-46c9-9df8-252cb4493f3b None None] PATH_INFO: `/` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:55:22.650174 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 712/1412] 10.222.0.27 () {54 vars in 918 bytes} [Tue Aug 30 01:55:22 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 01:55:22.654859 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eda132b1-f071-4519-97f1-e3c2dbe53192 None None] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:55:22.655147 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eda132b1-f071-4519-97f1-e3c2dbe53192 None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:55:22.655422 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eda132b1-f071-4519-97f1-e3c2dbe53192 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:55:22.728845 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:55:22.732024 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.auth.core [None req-eda132b1-f071-4519-97f1-e3c2dbe53192 None None] MFA Rules not processed for user `d64f2e71361644589bec8a06e6eceed1`. Rule list: `[]` (Enabled: `True`). {{(pid=81578) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 01:55:22.733246 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-eda132b1-f071-4519-97f1-e3c2dbe53192 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:55:22.856216 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 701/1413] 10.222.0.27 () {58 vars in 1015 bytes} [Tue Aug 30 01:55:22 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 202 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 01:55:25.248308 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da392a4b-2c68-4581-9432-921c5e7189d2 None None] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:55:25.248829 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da392a4b-2c68-4581-9432-921c5e7189d2 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:55:25.248829 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da392a4b-2c68-4581-9432-921c5e7189d2 None None] PATH_INFO: `/` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:55:25.250136 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 713/1414] 127.0.0.1 () {54 vars in 917 bytes} [Tue Aug 30 01:55:25 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 01:55:25.255258 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-597f3d28-6b83-4d2d-94b0-57a4ccedc733 None None] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:55:25.255684 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-597f3d28-6b83-4d2d-94b0-57a4ccedc733 None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:55:25.255790 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-597f3d28-6b83-4d2d-94b0-57a4ccedc733 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:55:25.320588 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.auth.core [None req-597f3d28-6b83-4d2d-94b0-57a4ccedc733 None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81578) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 01:55:25.321822 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-597f3d28-6b83-4d2d-94b0-57a4ccedc733 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:55:25.376390 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:55:25.418259 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 702/1415] 127.0.0.1 () {58 vars in 1014 bytes} [Tue Aug 30 01:55:25 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 164 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 01:55:25.711109 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da392a4b-2c68-4581-9432-921c5e7189d2 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:55:25.712606 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-da392a4b-2c68-4581-9432-921c5e7189d2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:55:25.857702 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da392a4b-2c68-4581-9432-921c5e7189d2 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:55:25.928770 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:55:25.969275 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-893bd676-73cb-4626-a41e-381f27addf4f service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd64f2e71361644589bec8a06e6eceed1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4562f3951fbe465280073e60a11e44e2', 'project_domain_id': 'default', 'roles': ['member', 'service', 'admin', 'ResellerAdmin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:55:25.969834 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-893bd676-73cb-4626-a41e-381f27addf4f service nova] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:55:25.970110 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-893bd676-73cb-4626-a41e-381f27addf4f service nova] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:55:25.970377 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-893bd676-73cb-4626-a41e-381f27addf4f service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:55:25.972212 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-893bd676-73cb-4626-a41e-381f27addf4f service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:55:26.063810 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-893bd676-73cb-4626-a41e-381f27addf4f service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:55:26.065434 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-893bd676-73cb-4626-a41e-381f27addf4f service nova] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:55:26.083021 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 714/1416] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Aug 30 01:55:25 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 372 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 01:55:26.982434 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9901031f-9579-4c17-a3cf-82f435802b83 None None] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:55:26.982954 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9901031f-9579-4c17-a3cf-82f435802b83 None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:55:26.982954 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9901031f-9579-4c17-a3cf-82f435802b83 None None] PATH_INFO: `/` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:55:26.984244 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 703/1417] 127.0.0.1 () {54 vars in 901 bytes} [Tue Aug 30 01:55:26 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 01:55:26.989281 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5d19283-05da-4083-8e6b-355da60158c2 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:55:26.989730 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5d19283-05da-4083-8e6b-355da60158c2 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:55:26.989863 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5d19283-05da-4083-8e6b-355da60158c2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:55:27.064889 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-f5d19283-05da-4083-8e6b-355da60158c2 None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 01:55:27.066260 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-f5d19283-05da-4083-8e6b-355da60158c2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:55:27.093265 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:55:27.163288 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 715/1418] 127.0.0.1 () {58 vars in 998 bytes} [Tue Aug 30 01:55:26 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 175 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 01:55:27.313055 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9901031f-9579-4c17-a3cf-82f435802b83 None None] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:55:27.314598 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-9901031f-9579-4c17-a3cf-82f435802b83 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:55:27.459535 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9901031f-9579-4c17-a3cf-82f435802b83 None None] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:55:27.499723 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:55:27.572411 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2404812-ac91-4c91-a543-cca15a8f3829 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd64f2e71361644589bec8a06e6eceed1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4562f3951fbe465280073e60a11e44e2', 'project_domain_id': 'default', 'roles': ['member', 'service', 'admin', 'ResellerAdmin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:55:27.572927 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2404812-ac91-4c91-a543-cca15a8f3829 service nova] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:55:27.573206 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2404812-ac91-4c91-a543-cca15a8f3829 service nova] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:55:27.573493 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2404812-ac91-4c91-a543-cca15a8f3829 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:55:27.575171 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-c2404812-ac91-4c91-a543-cca15a8f3829 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:55:27.661106 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2404812-ac91-4c91-a543-cca15a8f3829 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:55:27.662698 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2404812-ac91-4c91-a543-cca15a8f3829 service nova] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:55:27.680909 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 704/1419] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Aug 30 01:55:27 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 368 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 01:55:28.567569 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9708ca18-fea9-4867-9fac-efa1b6026bcc None None] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:55:28.568131 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9708ca18-fea9-4867-9fac-efa1b6026bcc None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:55:28.568131 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9708ca18-fea9-4867-9fac-efa1b6026bcc None None] PATH_INFO: `/` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:55:28.569510 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 716/1420] 127.0.0.1 () {54 vars in 901 bytes} [Tue Aug 30 01:55:28 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 01:55:28.574972 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4748143a-b1ed-4f8a-8d29-c159be346e97 None None] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:55:28.575370 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4748143a-b1ed-4f8a-8d29-c159be346e97 None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:55:28.575519 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4748143a-b1ed-4f8a-8d29-c159be346e97 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:55:28.629592 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:55:28.651252 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.auth.core [None req-4748143a-b1ed-4f8a-8d29-c159be346e97 None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81578) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 01:55:28.652663 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-4748143a-b1ed-4f8a-8d29-c159be346e97 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:55:28.747879 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 705/1421] 127.0.0.1 () {58 vars in 998 bytes} [Tue Aug 30 01:55:28 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 174 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 01:55:28.906508 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9708ca18-fea9-4867-9fac-efa1b6026bcc None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:55:28.907936 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-9708ca18-fea9-4867-9fac-efa1b6026bcc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:55:29.052367 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9708ca18-fea9-4867-9fac-efa1b6026bcc None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:55:29.065193 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:55:29.166122 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a390769-8e39-4008-b557-db8de39f4c15 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd64f2e71361644589bec8a06e6eceed1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4562f3951fbe465280073e60a11e44e2', 'project_domain_id': 'default', 'roles': ['member', 'service', 'admin', 'ResellerAdmin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:55:29.166635 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a390769-8e39-4008-b557-db8de39f4c15 service nova] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:55:29.166909 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a390769-8e39-4008-b557-db8de39f4c15 service nova] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:55:29.167174 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a390769-8e39-4008-b557-db8de39f4c15 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:55:29.169291 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-7a390769-8e39-4008-b557-db8de39f4c15 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:55:29.258891 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a390769-8e39-4008-b557-db8de39f4c15 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:55:29.260484 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a390769-8e39-4008-b557-db8de39f4c15 service nova] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:55:29.279311 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 717/1422] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Aug 30 01:55:28 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 373 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 01:55:30.153327 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1184ef85-7820-4521-a071-a3dbd6591d59 None None] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:55:30.153925 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1184ef85-7820-4521-a071-a3dbd6591d59 None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:55:30.153925 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1184ef85-7820-4521-a071-a3dbd6591d59 None None] PATH_INFO: `/` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:55:30.155308 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 706/1423] 127.0.0.1 () {54 vars in 901 bytes} [Tue Aug 30 01:55:30 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 01:55:30.160249 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55cedd7d-2d1c-4931-8b67-f98ef8476f50 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:55:30.160629 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55cedd7d-2d1c-4931-8b67-f98ef8476f50 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:55:30.160792 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55cedd7d-2d1c-4931-8b67-f98ef8476f50 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:55:30.181375 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:55:30.239919 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-55cedd7d-2d1c-4931-8b67-f98ef8476f50 None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 01:55:30.241065 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-55cedd7d-2d1c-4931-8b67-f98ef8476f50 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:55:30.335450 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 718/1424] 127.0.0.1 () {58 vars in 998 bytes} [Tue Aug 30 01:55:30 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 176 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 01:55:30.489638 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1184ef85-7820-4521-a071-a3dbd6591d59 None None] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:55:30.491198 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-1184ef85-7820-4521-a071-a3dbd6591d59 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:55:30.606693 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:55:30.642779 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1184ef85-7820-4521-a071-a3dbd6591d59 None None] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:55:30.756076 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77376803-4b66-4f6b-a0f7-f13279af2b25 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd64f2e71361644589bec8a06e6eceed1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4562f3951fbe465280073e60a11e44e2', 'project_domain_id': 'default', 'roles': ['member', 'service', 'admin', 'ResellerAdmin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:55:30.756638 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77376803-4b66-4f6b-a0f7-f13279af2b25 service nova] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:55:30.756911 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77376803-4b66-4f6b-a0f7-f13279af2b25 service nova] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:55:30.757175 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77376803-4b66-4f6b-a0f7-f13279af2b25 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:55:30.758796 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-77376803-4b66-4f6b-a0f7-f13279af2b25 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:55:30.838214 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:55:30.852475 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77376803-4b66-4f6b-a0f7-f13279af2b25 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:55:30.854111 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77376803-4b66-4f6b-a0f7-f13279af2b25 service nova] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:55:30.874099 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 707/1425] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Aug 30 01:55:30 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 385 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 01:55:31.742658 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-685e0a31-8dc0-40fc-9c71-8420881edea9 None None] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:55:31.743113 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-685e0a31-8dc0-40fc-9c71-8420881edea9 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:55:31.743232 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-685e0a31-8dc0-40fc-9c71-8420881edea9 None None] PATH_INFO: `/` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:55:31.744535 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 719/1426] 127.0.0.1 () {54 vars in 901 bytes} [Tue Aug 30 01:55:31 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 01:55:31.750173 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5260c347-d3ab-45ab-911f-36003b207001 None None] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:55:31.750619 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5260c347-d3ab-45ab-911f-36003b207001 None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:55:31.750750 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5260c347-d3ab-45ab-911f-36003b207001 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:55:31.826278 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.auth.core [None req-5260c347-d3ab-45ab-911f-36003b207001 None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81578) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 01:55:31.827557 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-5260c347-d3ab-45ab-911f-36003b207001 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:55:31.923337 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 708/1427] 127.0.0.1 () {58 vars in 998 bytes} [Tue Aug 30 01:55:31 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 174 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 30 01:55:32.072734 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-685e0a31-8dc0-40fc-9c71-8420881edea9 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:55:32.074595 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-685e0a31-8dc0-40fc-9c71-8420881edea9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:55:32.165194 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:55:32.230258 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-685e0a31-8dc0-40fc-9c71-8420881edea9 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:55:32.344408 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57a87e47-2a72-4ad4-bf96-23ab53f3bff3 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd64f2e71361644589bec8a06e6eceed1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4562f3951fbe465280073e60a11e44e2', 'project_domain_id': 'default', 'roles': ['member', 'service', 'admin', 'ResellerAdmin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:55:32.345055 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57a87e47-2a72-4ad4-bf96-23ab53f3bff3 service nova] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:55:32.345333 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57a87e47-2a72-4ad4-bf96-23ab53f3bff3 service nova] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:55:32.345635 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57a87e47-2a72-4ad4-bf96-23ab53f3bff3 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:55:32.347494 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-57a87e47-2a72-4ad4-bf96-23ab53f3bff3 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:55:32.397143 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:55:32.438830 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57a87e47-2a72-4ad4-bf96-23ab53f3bff3 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:55:32.440405 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57a87e47-2a72-4ad4-bf96-23ab53f3bff3 service nova] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:55:32.458959 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 720/1428] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Aug 30 01:55:32 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 387 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 01:55:33.333088 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-033e2904-3a1d-494c-85d8-c58b98a6dc8e None None] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:55:33.333580 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-033e2904-3a1d-494c-85d8-c58b98a6dc8e None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:55:33.333689 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-033e2904-3a1d-494c-85d8-c58b98a6dc8e None None] PATH_INFO: `/` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:55:33.335003 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 709/1429] 127.0.0.1 () {54 vars in 901 bytes} [Tue Aug 30 01:55:33 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 01:55:33.340241 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddc57b79-57ab-4292-b2c1-eaf5fe02aaae None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:55:33.340710 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddc57b79-57ab-4292-b2c1-eaf5fe02aaae None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:55:33.340826 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddc57b79-57ab-4292-b2c1-eaf5fe02aaae None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:55:33.416718 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-ddc57b79-57ab-4292-b2c1-eaf5fe02aaae None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 01:55:33.418476 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-ddc57b79-57ab-4292-b2c1-eaf5fe02aaae None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:55:33.521133 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 721/1430] 127.0.0.1 () {58 vars in 998 bytes} [Tue Aug 30 01:55:33 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 182 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 01:55:33.674577 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-033e2904-3a1d-494c-85d8-c58b98a6dc8e None None] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:55:33.676001 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-033e2904-3a1d-494c-85d8-c58b98a6dc8e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:55:33.724250 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:55:33.825769 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-033e2904-3a1d-494c-85d8-c58b98a6dc8e None None] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:55:34.069215 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c06205c-4007-412c-8ad4-6852c1011996 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd64f2e71361644589bec8a06e6eceed1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4562f3951fbe465280073e60a11e44e2', 'project_domain_id': 'default', 'roles': ['member', 'service', 'admin', 'ResellerAdmin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:55:34.069853 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c06205c-4007-412c-8ad4-6852c1011996 service nova] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:55:34.070134 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c06205c-4007-412c-8ad4-6852c1011996 service nova] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:55:34.070403 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c06205c-4007-412c-8ad4-6852c1011996 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:55:34.072015 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-3c06205c-4007-412c-8ad4-6852c1011996 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:55:34.080137 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:55:34.162145 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c06205c-4007-412c-8ad4-6852c1011996 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:55:34.163751 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c06205c-4007-412c-8ad4-6852c1011996 service nova] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:55:34.185789 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 710/1431] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Aug 30 01:55:33 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 512 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 01:55:35.090341 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a81d560-3a8d-4521-a6cb-8dadf000cfa6 None None] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:55:35.090909 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a81d560-3a8d-4521-a6cb-8dadf000cfa6 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:55:35.090909 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a81d560-3a8d-4521-a6cb-8dadf000cfa6 None None] PATH_INFO: `/` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:55:35.092254 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 722/1432] 127.0.0.1 () {54 vars in 901 bytes} [Tue Aug 30 01:55:35 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 01:55:35.097522 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3e1db12-cbf7-4df3-adb8-e7d4f74ea334 None None] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:55:35.097946 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3e1db12-cbf7-4df3-adb8-e7d4f74ea334 None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:55:35.098097 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3e1db12-cbf7-4df3-adb8-e7d4f74ea334 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:55:35.172873 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.auth.core [None req-b3e1db12-cbf7-4df3-adb8-e7d4f74ea334 None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81578) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 01:55:35.174587 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-b3e1db12-cbf7-4df3-adb8-e7d4f74ea334 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:55:35.240353 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:55:35.270596 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 711/1433] 127.0.0.1 () {58 vars in 998 bytes} [Tue Aug 30 01:55:35 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 174 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 01:55:35.421200 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a81d560-3a8d-4521-a6cb-8dadf000cfa6 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:55:35.422876 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-3a81d560-3a8d-4521-a6cb-8dadf000cfa6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:55:35.440768 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:55:35.569353 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a81d560-3a8d-4521-a6cb-8dadf000cfa6 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:55:35.656047 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:55:35.681512 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16039e72-a6fe-45af-b97e-c6b6be531699 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd64f2e71361644589bec8a06e6eceed1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4562f3951fbe465280073e60a11e44e2', 'project_domain_id': 'default', 'roles': ['member', 'service', 'admin', 'ResellerAdmin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:55:35.682118 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16039e72-a6fe-45af-b97e-c6b6be531699 service nova] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:55:35.682396 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16039e72-a6fe-45af-b97e-c6b6be531699 service nova] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:55:35.683416 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16039e72-a6fe-45af-b97e-c6b6be531699 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:55:35.685702 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-16039e72-a6fe-45af-b97e-c6b6be531699 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:55:35.777958 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16039e72-a6fe-45af-b97e-c6b6be531699 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:55:35.779537 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16039e72-a6fe-45af-b97e-c6b6be531699 service nova] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:55:35.798807 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 723/1434] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Aug 30 01:55:35 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 378 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 01:55:36.728747 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e20f9da-834a-4892-b633-e82a3a7bd502 None None] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:55:36.729245 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e20f9da-834a-4892-b633-e82a3a7bd502 None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:55:36.729357 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e20f9da-834a-4892-b633-e82a3a7bd502 None None] PATH_INFO: `/` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:55:36.730827 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 712/1435] 127.0.0.1 () {54 vars in 901 bytes} [Tue Aug 30 01:55:36 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 01:55:36.735819 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63ad0232-c9d0-4f3e-b9d7-65863be9f0a3 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:55:36.736231 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63ad0232-c9d0-4f3e-b9d7-65863be9f0a3 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:55:36.736345 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63ad0232-c9d0-4f3e-b9d7-65863be9f0a3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:55:36.810679 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-63ad0232-c9d0-4f3e-b9d7-65863be9f0a3 None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 01:55:36.811888 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-63ad0232-c9d0-4f3e-b9d7-65863be9f0a3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:55:36.853225 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:55:36.909198 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 724/1436] 127.0.0.1 () {58 vars in 998 bytes} [Tue Aug 30 01:55:36 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 174 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 01:55:37.059142 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e20f9da-834a-4892-b633-e82a3a7bd502 None None] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:55:37.060672 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-8e20f9da-834a-4892-b633-e82a3a7bd502 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:55:37.215833 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e20f9da-834a-4892-b633-e82a3a7bd502 None None] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:55:37.269682 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:55:37.326391 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3433f662-ad72-43ca-b1d0-cc3567560846 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd64f2e71361644589bec8a06e6eceed1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4562f3951fbe465280073e60a11e44e2', 'project_domain_id': 'default', 'roles': ['member', 'service', 'admin', 'ResellerAdmin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:55:37.326989 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3433f662-ad72-43ca-b1d0-cc3567560846 service nova] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:55:37.327264 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3433f662-ad72-43ca-b1d0-cc3567560846 service nova] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:55:37.327525 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3433f662-ad72-43ca-b1d0-cc3567560846 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:55:37.329132 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-3433f662-ad72-43ca-b1d0-cc3567560846 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:55:37.416113 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3433f662-ad72-43ca-b1d0-cc3567560846 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:55:37.418340 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3433f662-ad72-43ca-b1d0-cc3567560846 service nova] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:55:37.436536 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 713/1437] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Aug 30 01:55:37 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 379 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 01:55:38.300961 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1eaddf64-961a-496d-b29a-1e097442c5c1 None None] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:55:38.301524 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1eaddf64-961a-496d-b29a-1e097442c5c1 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:55:38.301623 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1eaddf64-961a-496d-b29a-1e097442c5c1 None None] PATH_INFO: `/` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:55:38.302896 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 725/1438] 127.0.0.1 () {54 vars in 901 bytes} [Tue Aug 30 01:55:38 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 01:55:38.308319 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c2ec6cf-cb7c-4690-9be3-59f9df84c5cb None None] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:55:38.308756 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c2ec6cf-cb7c-4690-9be3-59f9df84c5cb None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:55:38.308853 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c2ec6cf-cb7c-4690-9be3-59f9df84c5cb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:55:38.384758 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.auth.core [None req-3c2ec6cf-cb7c-4690-9be3-59f9df84c5cb None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81578) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 01:55:38.386615 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-3c2ec6cf-cb7c-4690-9be3-59f9df84c5cb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:55:38.388619 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:55:38.482442 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 714/1439] 127.0.0.1 () {58 vars in 998 bytes} [Tue Aug 30 01:55:38 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 176 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 01:55:38.632347 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1eaddf64-961a-496d-b29a-1e097442c5c1 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:55:38.633866 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-1eaddf64-961a-496d-b29a-1e097442c5c1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:55:38.782517 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1eaddf64-961a-496d-b29a-1e097442c5c1 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:55:38.809902 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:55:38.894524 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-686ffd74-be03-418c-8fe2-538ecea748d9 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd64f2e71361644589bec8a06e6eceed1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4562f3951fbe465280073e60a11e44e2', 'project_domain_id': 'default', 'roles': ['member', 'service', 'admin', 'ResellerAdmin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:55:38.895120 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-686ffd74-be03-418c-8fe2-538ecea748d9 service nova] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:55:38.895395 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-686ffd74-be03-418c-8fe2-538ecea748d9 service nova] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:55:38.895661 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-686ffd74-be03-418c-8fe2-538ecea748d9 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:55:38.897225 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-686ffd74-be03-418c-8fe2-538ecea748d9 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:55:38.989660 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-686ffd74-be03-418c-8fe2-538ecea748d9 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:55:38.991163 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-686ffd74-be03-418c-8fe2-538ecea748d9 service nova] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:55:39.009283 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 726/1440] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Aug 30 01:55:38 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 377 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 01:55:40.846536 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19eab300-c9ba-423e-8026-36472613a164 None None] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:55:40.847101 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19eab300-c9ba-423e-8026-36472613a164 None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:55:40.847101 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19eab300-c9ba-423e-8026-36472613a164 None None] PATH_INFO: `/` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:55:40.848395 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 715/1441] 127.0.0.1 () {54 vars in 917 bytes} [Tue Aug 30 01:55:40 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 01:55:40.853557 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0aa0cba4-6617-4c0c-9810-9dec99c7482c None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:55:40.853979 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0aa0cba4-6617-4c0c-9810-9dec99c7482c None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:55:40.854100 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0aa0cba4-6617-4c0c-9810-9dec99c7482c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:55:40.894581 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:55:40.920445 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-0aa0cba4-6617-4c0c-9810-9dec99c7482c None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 01:55:40.921695 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-0aa0cba4-6617-4c0c-9810-9dec99c7482c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:55:41.026772 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 727/1442] 127.0.0.1 () {58 vars in 1014 bytes} [Tue Aug 30 01:55:40 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 174 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 01:55:41.314962 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19eab300-c9ba-423e-8026-36472613a164 None None] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:55:41.317429 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-19eab300-c9ba-423e-8026-36472613a164 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:55:41.424057 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19eab300-c9ba-423e-8026-36472613a164 None None] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:55:41.457331 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:55:41.497885 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c3e5f11-e259-4b2d-846a-1deecd0073fc service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f3ea1580c56f4814a92bbb2acf542905', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4562f3951fbe465280073e60a11e44e2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:55:41.498484 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c3e5f11-e259-4b2d-846a-1deecd0073fc service neutron] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:55:41.498759 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c3e5f11-e259-4b2d-846a-1deecd0073fc service neutron] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:55:41.499022 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c3e5f11-e259-4b2d-846a-1deecd0073fc service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:55:41.500613 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-5c3e5f11-e259-4b2d-846a-1deecd0073fc service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:55:41.608556 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c3e5f11-e259-4b2d-846a-1deecd0073fc service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:55:41.610026 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c3e5f11-e259-4b2d-846a-1deecd0073fc service neutron] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:55:41.628320 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 716/1443] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Aug 30 01:55:41 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 314 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 01:55:43.751089 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36bf43a1-f4cb-4d88-a7e1-95ed7ca55a1b None None] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:55:43.751632 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36bf43a1-f4cb-4d88-a7e1-95ed7ca55a1b None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:55:43.751632 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36bf43a1-f4cb-4d88-a7e1-95ed7ca55a1b None None] PATH_INFO: `/` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:55:43.752982 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 728/1444] 127.0.0.1 () {54 vars in 917 bytes} [Tue Aug 30 01:55:43 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 01:55:43.758144 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8a311f4-6441-4cf5-a6a1-6de13de7239b None None] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:55:43.758581 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8a311f4-6441-4cf5-a6a1-6de13de7239b None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:55:43.758702 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8a311f4-6441-4cf5-a6a1-6de13de7239b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:55:43.825655 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.auth.core [None req-c8a311f4-6441-4cf5-a6a1-6de13de7239b None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81578) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 01:55:43.826846 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-c8a311f4-6441-4cf5-a6a1-6de13de7239b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:55:43.839180 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:55:43.920492 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 717/1445] 127.0.0.1 () {58 vars in 1014 bytes} [Tue Aug 30 01:55:43 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 163 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 01:55:44.208963 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36bf43a1-f4cb-4d88-a7e1-95ed7ca55a1b None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:55:44.210526 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-36bf43a1-f4cb-4d88-a7e1-95ed7ca55a1b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:55:44.313353 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36bf43a1-f4cb-4d88-a7e1-95ed7ca55a1b None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:55:44.325606 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:55:44.388003 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3e4a642-7e20-4501-8b9c-adbd857b4b18 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f3ea1580c56f4814a92bbb2acf542905', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4562f3951fbe465280073e60a11e44e2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:55:44.388551 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3e4a642-7e20-4501-8b9c-adbd857b4b18 service neutron] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:55:44.388821 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3e4a642-7e20-4501-8b9c-adbd857b4b18 service neutron] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:55:44.389085 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3e4a642-7e20-4501-8b9c-adbd857b4b18 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:55:44.390716 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-b3e4a642-7e20-4501-8b9c-adbd857b4b18 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:55:44.506646 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3e4a642-7e20-4501-8b9c-adbd857b4b18 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:55:44.508081 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3e4a642-7e20-4501-8b9c-adbd857b4b18 service neutron] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:55:44.526553 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 729/1446] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Aug 30 01:55:44 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 318 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 01:55:46.902358 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e705262-d924-4367-84c8-4ba23c092a33 None None] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:55:46.902890 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e705262-d924-4367-84c8-4ba23c092a33 None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:55:46.902890 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e705262-d924-4367-84c8-4ba23c092a33 None None] PATH_INFO: `/` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:55:46.904138 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 718/1447] 127.0.0.1 () {54 vars in 917 bytes} [Tue Aug 30 01:55:46 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 01:55:46.909368 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a85b88ed-18dc-46a7-a0a7-149bf094fb5d None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:55:46.909796 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a85b88ed-18dc-46a7-a0a7-149bf094fb5d None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:55:46.909958 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a85b88ed-18dc-46a7-a0a7-149bf094fb5d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:55:46.958662 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:55:46.984958 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-a85b88ed-18dc-46a7-a0a7-149bf094fb5d None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 01:55:46.986216 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-a85b88ed-18dc-46a7-a0a7-149bf094fb5d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:55:47.081288 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 730/1448] 127.0.0.1 () {58 vars in 1014 bytes} [Tue Aug 30 01:55:46 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 173 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 01:55:47.368435 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e705262-d924-4367-84c8-4ba23c092a33 None None] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:55:47.372803 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-7e705262-d924-4367-84c8-4ba23c092a33 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:55:47.474063 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e705262-d924-4367-84c8-4ba23c092a33 None None] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:55:47.515926 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:55:47.549593 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab01a960-d878-40c4-ac82-3095423199f9 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f3ea1580c56f4814a92bbb2acf542905', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4562f3951fbe465280073e60a11e44e2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:55:47.550456 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab01a960-d878-40c4-ac82-3095423199f9 service neutron] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:55:47.550998 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab01a960-d878-40c4-ac82-3095423199f9 service neutron] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:55:47.551573 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab01a960-d878-40c4-ac82-3095423199f9 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:55:47.553233 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-ab01a960-d878-40c4-ac82-3095423199f9 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:55:47.670738 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab01a960-d878-40c4-ac82-3095423199f9 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:55:47.672066 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab01a960-d878-40c4-ac82-3095423199f9 service neutron] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:55:47.690489 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 719/1449] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Aug 30 01:55:47 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 323 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 01:55:49.870949 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba755c15-e892-483c-899d-067f86ae8318 None None] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:55:49.871525 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba755c15-e892-483c-899d-067f86ae8318 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:55:49.871525 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba755c15-e892-483c-899d-067f86ae8318 None None] PATH_INFO: `/` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:55:49.872847 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 731/1450] 127.0.0.1 () {54 vars in 917 bytes} [Tue Aug 30 01:55:49 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 01:55:49.878331 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fe2dd4e-25e9-423a-9bcf-e633c9911fba None None] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:55:49.878776 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fe2dd4e-25e9-423a-9bcf-e633c9911fba None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:55:49.878891 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fe2dd4e-25e9-423a-9bcf-e633c9911fba None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:55:49.943430 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.auth.core [None req-0fe2dd4e-25e9-423a-9bcf-e633c9911fba None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81578) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 01:55:49.944629 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-0fe2dd4e-25e9-423a-9bcf-e633c9911fba None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:55:49.965542 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:55:50.045354 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 720/1451] 127.0.0.1 () {58 vars in 1014 bytes} [Tue Aug 30 01:55:49 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 168 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 01:55:50.330232 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba755c15-e892-483c-899d-067f86ae8318 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:55:50.331769 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-ba755c15-e892-483c-899d-067f86ae8318 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:55:50.439367 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba755c15-e892-483c-899d-067f86ae8318 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:55:50.459083 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:55:50.514497 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3153472-e833-4cfd-b070-f6436eef5d18 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f3ea1580c56f4814a92bbb2acf542905', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4562f3951fbe465280073e60a11e44e2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:55:50.515101 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3153472-e833-4cfd-b070-f6436eef5d18 service neutron] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:55:50.515378 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3153472-e833-4cfd-b070-f6436eef5d18 service neutron] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:55:50.515645 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3153472-e833-4cfd-b070-f6436eef5d18 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:55:50.517273 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-d3153472-e833-4cfd-b070-f6436eef5d18 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:55:50.631477 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3153472-e833-4cfd-b070-f6436eef5d18 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:55:50.632721 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3153472-e833-4cfd-b070-f6436eef5d18 service neutron] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:55:50.651962 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 732/1452] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Aug 30 01:55:50 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 322 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 01:55:52.850421 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e83990a-b6c3-403b-8c30-e8d9a078af59 None None] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:55:52.850983 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e83990a-b6c3-403b-8c30-e8d9a078af59 None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:55:52.850983 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e83990a-b6c3-403b-8c30-e8d9a078af59 None None] PATH_INFO: `/` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:55:52.852308 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 721/1453] 127.0.0.1 () {54 vars in 917 bytes} [Tue Aug 30 01:55:52 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 01:55:52.858262 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b8bead3-e399-4b4c-9f4c-56e06ee11db8 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:55:52.858678 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b8bead3-e399-4b4c-9f4c-56e06ee11db8 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:55:52.858820 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b8bead3-e399-4b4c-9f4c-56e06ee11db8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:55:52.921860 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:55:52.924365 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-7b8bead3-e399-4b4c-9f4c-56e06ee11db8 None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 01:55:52.925598 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-7b8bead3-e399-4b4c-9f4c-56e06ee11db8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:55:53.032240 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 733/1454] 127.0.0.1 () {58 vars in 1014 bytes} [Tue Aug 30 01:55:52 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 175 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 01:55:53.317411 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e83990a-b6c3-403b-8c30-e8d9a078af59 None None] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:55:53.319004 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-6e83990a-b6c3-403b-8c30-e8d9a078af59 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:55:53.423748 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e83990a-b6c3-403b-8c30-e8d9a078af59 None None] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:55:53.471635 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:55:53.496056 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62f379cb-9330-4ec0-a129-c6f88539d033 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f3ea1580c56f4814a92bbb2acf542905', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4562f3951fbe465280073e60a11e44e2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:55:53.496660 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62f379cb-9330-4ec0-a129-c6f88539d033 service neutron] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:55:53.496939 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62f379cb-9330-4ec0-a129-c6f88539d033 service neutron] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:55:53.497209 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62f379cb-9330-4ec0-a129-c6f88539d033 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:55:53.498865 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-62f379cb-9330-4ec0-a129-c6f88539d033 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:55:53.606848 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62f379cb-9330-4ec0-a129-c6f88539d033 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:55:53.608194 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62f379cb-9330-4ec0-a129-c6f88539d033 service neutron] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:55:53.627364 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 722/1455] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Aug 30 01:55:53 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 310 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 01:55:55.594008 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fda32533-508a-46ce-846b-2c2a37633423 None None] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:55:55.594571 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fda32533-508a-46ce-846b-2c2a37633423 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:55:55.594571 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fda32533-508a-46ce-846b-2c2a37633423 None None] PATH_INFO: `/` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:55:55.595901 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 734/1456] 127.0.0.1 () {54 vars in 917 bytes} [Tue Aug 30 01:55:55 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 01:55:55.601377 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fce2f082-f255-4b7f-89c9-7f0c8853ed12 None None] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:55:55.601857 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fce2f082-f255-4b7f-89c9-7f0c8853ed12 None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:55:55.601955 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fce2f082-f255-4b7f-89c9-7f0c8853ed12 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:55:55.683435 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.auth.core [None req-fce2f082-f255-4b7f-89c9-7f0c8853ed12 None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81578) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 01:55:55.684648 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-fce2f082-f255-4b7f-89c9-7f0c8853ed12 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:55:55.713663 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:55:55.781478 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 723/1457] 127.0.0.1 () {58 vars in 1014 bytes} [Tue Aug 30 01:55:55 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 181 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 01:55:56.063225 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fda32533-508a-46ce-846b-2c2a37633423 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:55:56.064759 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-fda32533-508a-46ce-846b-2c2a37633423 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:55:56.167439 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fda32533-508a-46ce-846b-2c2a37633423 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:55:56.194731 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:55:56.241304 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f2ee057-d08d-41a5-85de-4dd324a544ec service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f3ea1580c56f4814a92bbb2acf542905', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4562f3951fbe465280073e60a11e44e2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:55:56.242133 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f2ee057-d08d-41a5-85de-4dd324a544ec service neutron] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:55:56.242755 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f2ee057-d08d-41a5-85de-4dd324a544ec service neutron] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:55:56.243031 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f2ee057-d08d-41a5-85de-4dd324a544ec service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:55:56.244785 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-8f2ee057-d08d-41a5-85de-4dd324a544ec service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:55:56.352970 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f2ee057-d08d-41a5-85de-4dd324a544ec service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:55:56.354436 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f2ee057-d08d-41a5-85de-4dd324a544ec service neutron] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:55:56.373010 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 735/1458] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Aug 30 01:55:56 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 310 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 01:55:58.349121 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-390d36bb-ead9-4276-9f51-768eac106f68 None None] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:55:58.349689 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-390d36bb-ead9-4276-9f51-768eac106f68 None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:55:58.349689 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-390d36bb-ead9-4276-9f51-768eac106f68 None None] PATH_INFO: `/` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:55:58.351046 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 724/1459] 127.0.0.1 () {54 vars in 917 bytes} [Tue Aug 30 01:55:58 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 01:55:58.357033 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2dcae56-342a-4b6f-acb9-f9d58c481785 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:55:58.357445 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2dcae56-342a-4b6f-acb9-f9d58c481785 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:55:58.357605 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2dcae56-342a-4b6f-acb9-f9d58c481785 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:55:58.423063 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-d2dcae56-342a-4b6f-acb9-f9d58c481785 None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 01:55:58.424256 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-d2dcae56-342a-4b6f-acb9-f9d58c481785 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:55:58.433181 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:55:58.521863 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 736/1460] 127.0.0.1 () {58 vars in 1014 bytes} [Tue Aug 30 01:55:58 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 166 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 01:55:58.809078 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-390d36bb-ead9-4276-9f51-768eac106f68 None None] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:55:58.810757 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-390d36bb-ead9-4276-9f51-768eac106f68 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:55:58.920272 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-390d36bb-ead9-4276-9f51-768eac106f68 None None] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:55:58.988967 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:55:59.011639 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80f5ce5f-b7b6-497e-8bcc-e26acfea0e81 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f3ea1580c56f4814a92bbb2acf542905', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4562f3951fbe465280073e60a11e44e2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:55:59.012531 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80f5ce5f-b7b6-497e-8bcc-e26acfea0e81 service neutron] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:55:59.013784 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80f5ce5f-b7b6-497e-8bcc-e26acfea0e81 service neutron] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:55:59.013784 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80f5ce5f-b7b6-497e-8bcc-e26acfea0e81 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:55:59.014664 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-80f5ce5f-b7b6-497e-8bcc-e26acfea0e81 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:55:59.136124 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80f5ce5f-b7b6-497e-8bcc-e26acfea0e81 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:55:59.137657 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80f5ce5f-b7b6-497e-8bcc-e26acfea0e81 service neutron] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:55:59.156549 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 725/1461] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Aug 30 01:55:58 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 348 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 01:56:01.202361 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca2bc445-9b57-451c-afc1-be680b64f932 None None] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:56:01.202954 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca2bc445-9b57-451c-afc1-be680b64f932 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:56:01.202954 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca2bc445-9b57-451c-afc1-be680b64f932 None None] PATH_INFO: `/` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:56:01.204244 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 737/1462] 127.0.0.1 () {54 vars in 917 bytes} [Tue Aug 30 01:56:01 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 01:56:01.209514 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78d70cf2-640a-4b57-a64f-78733d890abd None None] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:56:01.210049 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78d70cf2-640a-4b57-a64f-78733d890abd None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:56:01.210540 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78d70cf2-640a-4b57-a64f-78733d890abd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:56:01.281477 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.auth.core [None req-78d70cf2-640a-4b57-a64f-78733d890abd None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81578) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 01:56:01.282709 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-78d70cf2-640a-4b57-a64f-78733d890abd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:56:01.321154 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:56:01.378592 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 726/1463] 127.0.0.1 () {58 vars in 1014 bytes} [Tue Aug 30 01:56:01 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 170 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 01:56:01.661997 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca2bc445-9b57-451c-afc1-be680b64f932 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:56:01.663563 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-ca2bc445-9b57-451c-afc1-be680b64f932 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:56:01.771354 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca2bc445-9b57-451c-afc1-be680b64f932 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:56:01.810280 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:56:01.846562 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3ade89c-637f-48f9-8e83-136610404f07 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f3ea1580c56f4814a92bbb2acf542905', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4562f3951fbe465280073e60a11e44e2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:56:01.847165 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3ade89c-637f-48f9-8e83-136610404f07 service neutron] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:56:01.847445 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3ade89c-637f-48f9-8e83-136610404f07 service neutron] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:56:01.847712 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3ade89c-637f-48f9-8e83-136610404f07 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:56:01.849312 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-e3ade89c-637f-48f9-8e83-136610404f07 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:56:01.956443 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3ade89c-637f-48f9-8e83-136610404f07 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:56:01.957927 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3ade89c-637f-48f9-8e83-136610404f07 service neutron] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:56:01.977233 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 738/1464] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Aug 30 01:56:01 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 316 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 30 01:56:03.898127 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7241240-7ac8-4c92-997f-cd185f1c1598 None None] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:56:03.898664 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7241240-7ac8-4c92-997f-cd185f1c1598 None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:56:03.898664 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7241240-7ac8-4c92-997f-cd185f1c1598 None None] PATH_INFO: `/` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:56:03.900012 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 727/1465] 127.0.0.1 () {54 vars in 917 bytes} [Tue Aug 30 01:56:03 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 01:56:03.905417 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b9144ed-79e9-41a7-b5a1-b58d72a596a5 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:56:03.905875 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b9144ed-79e9-41a7-b5a1-b58d72a596a5 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:56:03.906015 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b9144ed-79e9-41a7-b5a1-b58d72a596a5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:56:03.979996 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-7b9144ed-79e9-41a7-b5a1-b58d72a596a5 None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 01:56:03.981124 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-7b9144ed-79e9-41a7-b5a1-b58d72a596a5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:56:03.999614 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:56:04.078567 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 739/1466] 127.0.0.1 () {58 vars in 1014 bytes} [Tue Aug 30 01:56:03 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 174 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 01:56:04.364112 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7241240-7ac8-4c92-997f-cd185f1c1598 None None] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:56:04.365833 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-e7241240-7ac8-4c92-997f-cd185f1c1598 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:56:04.475580 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7241240-7ac8-4c92-997f-cd185f1c1598 None None] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:56:04.540382 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:56:04.546978 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40dfa703-2bb0-4c3d-9742-765731920680 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f3ea1580c56f4814a92bbb2acf542905', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4562f3951fbe465280073e60a11e44e2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:56:04.547575 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40dfa703-2bb0-4c3d-9742-765731920680 service neutron] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:56:04.553413 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40dfa703-2bb0-4c3d-9742-765731920680 service neutron] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:56:04.553413 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40dfa703-2bb0-4c3d-9742-765731920680 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:56:04.553413 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-40dfa703-2bb0-4c3d-9742-765731920680 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:56:04.662043 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40dfa703-2bb0-4c3d-9742-765731920680 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:56:04.663472 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40dfa703-2bb0-4c3d-9742-765731920680 service neutron] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:56:04.681289 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 728/1467] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Aug 30 01:56:04 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 318 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 01:56:06.599133 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e892caab-b835-4475-a8ed-47ec490c02d5 None None] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:56:06.599677 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e892caab-b835-4475-a8ed-47ec490c02d5 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:56:06.599677 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e892caab-b835-4475-a8ed-47ec490c02d5 None None] PATH_INFO: `/` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:56:06.601026 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 740/1468] 127.0.0.1 () {54 vars in 917 bytes} [Tue Aug 30 01:56:06 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 01:56:06.606350 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a6df80a-6d4f-435d-bf40-1b9b3901bbc6 None None] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:56:06.606786 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a6df80a-6d4f-435d-bf40-1b9b3901bbc6 None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:56:06.606884 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a6df80a-6d4f-435d-bf40-1b9b3901bbc6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:56:06.672735 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.auth.core [None req-8a6df80a-6d4f-435d-bf40-1b9b3901bbc6 None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81578) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 01:56:06.673982 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-8a6df80a-6d4f-435d-bf40-1b9b3901bbc6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:56:06.724617 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:56:06.772089 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 729/1469] 127.0.0.1 () {58 vars in 1014 bytes} [Tue Aug 30 01:56:06 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 167 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 01:56:07.060719 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e892caab-b835-4475-a8ed-47ec490c02d5 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:56:07.062292 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-e892caab-b835-4475-a8ed-47ec490c02d5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:56:07.168673 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e892caab-b835-4475-a8ed-47ec490c02d5 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:56:07.217197 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:56:07.249962 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2433cf04-2335-4b46-8a78-734d6f52e847 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f3ea1580c56f4814a92bbb2acf542905', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4562f3951fbe465280073e60a11e44e2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:56:07.250561 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2433cf04-2335-4b46-8a78-734d6f52e847 service neutron] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:56:07.250835 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2433cf04-2335-4b46-8a78-734d6f52e847 service neutron] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:56:07.251098 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2433cf04-2335-4b46-8a78-734d6f52e847 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:56:07.252736 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-2433cf04-2335-4b46-8a78-734d6f52e847 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:56:07.360989 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2433cf04-2335-4b46-8a78-734d6f52e847 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:56:07.362384 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2433cf04-2335-4b46-8a78-734d6f52e847 service neutron] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:56:07.381531 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 741/1470] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Aug 30 01:56:07 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 321 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 01:56:16.770315 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=22 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:56:17.379259 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=66 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:56:27.817813 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb27e687-a135-4660-9cea-6fe44533cd81 None None] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:56:27.818343 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb27e687-a135-4660-9cea-6fe44533cd81 None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:56:27.818343 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb27e687-a135-4660-9cea-6fe44533cd81 None None] PATH_INFO: `/` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:56:27.819698 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 730/1471] 10.222.0.27 () {54 vars in 927 bytes} [Tue Aug 30 01:56:27 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 01:56:27.825235 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a622b5a-e0cb-44bb-8972-4452d0967844 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:56:27.825712 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a622b5a-e0cb-44bb-8972-4452d0967844 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:56:27.825842 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a622b5a-e0cb-44bb-8972-4452d0967844 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:56:27.891890 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-5a622b5a-e0cb-44bb-8972-4452d0967844 None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 01:56:27.893100 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-5a622b5a-e0cb-44bb-8972-4452d0967844 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:56:27.988942 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 742/1472] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Aug 30 01:56:27 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 165 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 01:56:28.285174 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48b93922-279e-45ed-b14e-023413759a14 None None] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:56:28.285483 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48b93922-279e-45ed-b14e-023413759a14 None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:56:28.285775 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48b93922-279e-45ed-b14e-023413759a14 None None] PATH_INFO: `/` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:56:28.287097 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 731/1473] 10.222.0.27 () {54 vars in 967 bytes} [Tue Aug 30 01:56:28 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 01:56:28.292003 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b58ab96-f428-41ae-9f3f-c7a4f6658ee8 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:56:28.292318 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b58ab96-f428-41ae-9f3f-c7a4f6658ee8 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:56:28.292594 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b58ab96-f428-41ae-9f3f-c7a4f6658ee8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:56:28.485219 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-0b58ab96-f428-41ae-9f3f-c7a4f6658ee8 None None] MFA Rules not processed for user `3caff62093d748f48596574836e12e42`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 01:56:28.486479 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-0b58ab96-f428-41ae-9f3f-c7a4f6658ee8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:56:28.507220 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:56:28.564117 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 743/1474] 10.222.0.27 () {58 vars in 1064 bytes} [Tue Aug 30 01:56:28 2022] POST /identity/v3/auth/tokens => generated 3120 bytes in 273 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 01:56:28.572149 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48b93922-279e-45ed-b14e-023413759a14 None None] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:56:28.573714 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-48b93922-279e-45ed-b14e-023413759a14 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:56:28.674870 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48b93922-279e-45ed-b14e-023413759a14 None None] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:56:28.751680 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68bd3c95-0e93-4741-be0e-ac17e7c63bb5 service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3caff62093d748f48596574836e12e42', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4562f3951fbe465280073e60a11e44e2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:56:28.752281 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68bd3c95-0e93-4741-be0e-ac17e7c63bb5 service glance] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:56:28.752559 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68bd3c95-0e93-4741-be0e-ac17e7c63bb5 service glance] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:56:28.752828 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68bd3c95-0e93-4741-be0e-ac17e7c63bb5 service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:56:28.754446 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-68bd3c95-0e93-4741-be0e-ac17e7c63bb5 service glance] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:56:28.807785 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:56:28.861450 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68bd3c95-0e93-4741-be0e-ac17e7c63bb5 service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:56:28.862841 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68bd3c95-0e93-4741-be0e-ac17e7c63bb5 service glance] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:56:28.881373 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 732/1475] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Aug 30 01:56:28 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 310 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 01:56:30.886234 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdd48b40-6bb3-4ec7-8b08-332120aa33f9 None None] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:56:30.889623 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdd48b40-6bb3-4ec7-8b08-332120aa33f9 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:56:30.889623 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdd48b40-6bb3-4ec7-8b08-332120aa33f9 None None] PATH_INFO: `/` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:56:30.889623 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 744/1476] 10.222.0.27 () {54 vars in 927 bytes} [Tue Aug 30 01:56:30 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 01:56:30.893557 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d7fb054-f1c4-405d-bb6d-7ed99d5251c4 None None] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:56:30.893992 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d7fb054-f1c4-405d-bb6d-7ed99d5251c4 None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:56:30.894087 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d7fb054-f1c4-405d-bb6d-7ed99d5251c4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:56:30.962579 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.auth.core [None req-6d7fb054-f1c4-405d-bb6d-7ed99d5251c4 None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81578) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 01:56:30.963709 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-6d7fb054-f1c4-405d-bb6d-7ed99d5251c4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:56:31.060785 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 733/1477] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Aug 30 01:56:30 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 168 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 01:56:31.338721 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdd48b40-6bb3-4ec7-8b08-332120aa33f9 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:56:31.340240 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-bdd48b40-6bb3-4ec7-8b08-332120aa33f9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:56:31.466845 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdd48b40-6bb3-4ec7-8b08-332120aa33f9 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:56:31.515975 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:56:31.563287 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04b058cb-812f-41f7-a3cc-4e0e6f415f3e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:56:31.563895 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04b058cb-812f-41f7-a3cc-4e0e6f415f3e admin admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:56:31.564169 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04b058cb-812f-41f7-a3cc-4e0e6f415f3e admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:56:31.564433 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04b058cb-812f-41f7-a3cc-4e0e6f415f3e admin admin] PATH_INFO: `/v3/projects` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:56:31.565092 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04b058cb-812f-41f7-a3cc-4e0e6f415f3e admin admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:56:31.566848 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: WARNING py.warnings [None req-04b058cb-812f-41f7-a3cc-4e0e6f415f3e admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:list_projects": "(role:reader and system_scope:all) or (role:reader and domain_id:%(target.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Aug 30 01:56:31.566848 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: warnings.warn(msg) Aug 30 01:56:31.566848 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: Aug 30 01:56:31.567960 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04b058cb-812f-41f7-a3cc-4e0e6f415f3e admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:56:31.587210 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 745/1478] 10.222.0.27 () {56 vars in 1108 bytes} [Tue Aug 30 01:56:31 2022] GET /identity/v3/projects => generated 2553 bytes in 249 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 01:56:33.519960 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c8fe784-788c-4685-8199-ddea67c11645 None None] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:56:33.520531 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c8fe784-788c-4685-8199-ddea67c11645 None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:56:33.520531 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c8fe784-788c-4685-8199-ddea67c11645 None None] PATH_INFO: `/` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:56:33.521929 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 734/1479] 10.222.0.27 () {54 vars in 927 bytes} [Tue Aug 30 01:56:33 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 01:56:33.527333 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7acf8f35-1d62-4756-8b9e-eabba21c8242 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:56:33.527738 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7acf8f35-1d62-4756-8b9e-eabba21c8242 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:56:33.527874 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7acf8f35-1d62-4756-8b9e-eabba21c8242 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:56:33.592978 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-7acf8f35-1d62-4756-8b9e-eabba21c8242 None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 01:56:33.594211 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-7acf8f35-1d62-4756-8b9e-eabba21c8242 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:56:33.690488 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 746/1480] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Aug 30 01:56:33 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 164 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 01:56:33.990004 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c8fe784-788c-4685-8199-ddea67c11645 None None] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:56:33.991502 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-4c8fe784-788c-4685-8199-ddea67c11645 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:56:34.001798 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:56:34.138159 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c8fe784-788c-4685-8199-ddea67c11645 None None] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:56:34.216099 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:56:34.250320 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9c91013-2881-458a-8133-7ccd810e710a service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd64f2e71361644589bec8a06e6eceed1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4562f3951fbe465280073e60a11e44e2', 'project_domain_id': 'default', 'roles': ['member', 'service', 'admin', 'ResellerAdmin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:56:34.250853 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9c91013-2881-458a-8133-7ccd810e710a service nova] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:56:34.251132 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9c91013-2881-458a-8133-7ccd810e710a service nova] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:56:34.251401 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9c91013-2881-458a-8133-7ccd810e710a service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:56:34.253739 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-a9c91013-2881-458a-8133-7ccd810e710a service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:56:34.341651 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9c91013-2881-458a-8133-7ccd810e710a service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:56:34.343238 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9c91013-2881-458a-8133-7ccd810e710a service nova] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:56:34.361980 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 735/1481] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Aug 30 01:56:33 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 372 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 01:56:36.444809 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebb1e64e-7bec-41ab-8897-7c94f1f6ba54 None None] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:56:36.446079 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebb1e64e-7bec-41ab-8897-7c94f1f6ba54 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:56:36.446079 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebb1e64e-7bec-41ab-8897-7c94f1f6ba54 None None] PATH_INFO: `/` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:56:36.447345 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 747/1482] 10.222.0.27 () {54 vars in 927 bytes} [Tue Aug 30 01:56:36 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 01:56:36.452554 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d09fab6-ebed-47a0-9020-b5796fd485e7 None None] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:56:36.452966 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d09fab6-ebed-47a0-9020-b5796fd485e7 None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:56:36.453132 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d09fab6-ebed-47a0-9020-b5796fd485e7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:56:36.518363 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.auth.core [None req-3d09fab6-ebed-47a0-9020-b5796fd485e7 None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81578) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 01:56:36.519578 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-3d09fab6-ebed-47a0-9020-b5796fd485e7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:56:36.564259 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:56:36.614182 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 736/1483] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Aug 30 01:56:36 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 163 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 01:56:36.899211 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebb1e64e-7bec-41ab-8897-7c94f1f6ba54 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:56:36.900717 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-ebb1e64e-7bec-41ab-8897-7c94f1f6ba54 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:56:36.923582 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:56:37.002840 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebb1e64e-7bec-41ab-8897-7c94f1f6ba54 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:56:37.076738 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14315224-5772-48bc-a85c-e32b6669caf4 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f3ea1580c56f4814a92bbb2acf542905', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4562f3951fbe465280073e60a11e44e2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:56:37.077638 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14315224-5772-48bc-a85c-e32b6669caf4 service neutron] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:56:37.078058 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14315224-5772-48bc-a85c-e32b6669caf4 service neutron] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:56:37.078466 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14315224-5772-48bc-a85c-e32b6669caf4 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:56:37.080945 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-14315224-5772-48bc-a85c-e32b6669caf4 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:56:37.170573 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:56:37.205037 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14315224-5772-48bc-a85c-e32b6669caf4 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:56:37.206510 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14315224-5772-48bc-a85c-e32b6669caf4 service neutron] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:56:37.226009 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 748/1484] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Aug 30 01:56:36 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 327 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 01:56:39.194764 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d9c2eab-e6d1-4167-b58f-b91abe882807 None None] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:56:39.195403 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d9c2eab-e6d1-4167-b58f-b91abe882807 None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:56:39.195403 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d9c2eab-e6d1-4167-b58f-b91abe882807 None None] PATH_INFO: `/` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:56:39.196606 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 737/1485] 10.222.0.27 () {54 vars in 927 bytes} [Tue Aug 30 01:56:39 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 01:56:39.202076 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96d6236c-623a-4cc1-a1df-0e99f70b8002 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:56:39.202525 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96d6236c-623a-4cc1-a1df-0e99f70b8002 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:56:39.202634 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96d6236c-623a-4cc1-a1df-0e99f70b8002 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:56:39.275433 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-96d6236c-623a-4cc1-a1df-0e99f70b8002 None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 01:56:39.276641 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-96d6236c-623a-4cc1-a1df-0e99f70b8002 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:56:39.372227 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 749/1486] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Aug 30 01:56:39 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 171 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 01:56:39.663536 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d9c2eab-e6d1-4167-b58f-b91abe882807 None None] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:56:39.665697 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-8d9c2eab-e6d1-4167-b58f-b91abe882807 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:56:39.770099 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d9c2eab-e6d1-4167-b58f-b91abe882807 None None] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:56:39.841081 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:56:39.842845 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f1f9234-cec1-4024-8a19-8364f25ba639 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f3ea1580c56f4814a92bbb2acf542905', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4562f3951fbe465280073e60a11e44e2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:56:39.843449 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f1f9234-cec1-4024-8a19-8364f25ba639 service neutron] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:56:39.843733 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f1f9234-cec1-4024-8a19-8364f25ba639 service neutron] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:56:39.844001 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f1f9234-cec1-4024-8a19-8364f25ba639 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:56:39.845864 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-5f1f9234-cec1-4024-8a19-8364f25ba639 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:56:39.952627 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f1f9234-cec1-4024-8a19-8364f25ba639 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:56:39.954712 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f1f9234-cec1-4024-8a19-8364f25ba639 service neutron] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:56:39.976201 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 738/1487] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Aug 30 01:56:39 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 313 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 01:56:42.134692 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bc74a93-81cf-43ce-8056-a60d205aeb6c None None] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:56:42.135258 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bc74a93-81cf-43ce-8056-a60d205aeb6c None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:56:42.135258 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bc74a93-81cf-43ce-8056-a60d205aeb6c None None] PATH_INFO: `/` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:56:42.136569 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 750/1488] 10.222.0.27 () {54 vars in 927 bytes} [Tue Aug 30 01:56:42 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 30 01:56:42.141968 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74d269da-a992-4219-990f-3069da2bc6cb None None] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:56:42.142403 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74d269da-a992-4219-990f-3069da2bc6cb None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:56:42.142506 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74d269da-a992-4219-990f-3069da2bc6cb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:56:42.207828 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.auth.core [None req-74d269da-a992-4219-990f-3069da2bc6cb None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81578) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 01:56:42.209040 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-74d269da-a992-4219-990f-3069da2bc6cb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:56:42.269961 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:56:42.313090 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 739/1489] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Aug 30 01:56:42 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 172 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 01:56:42.626426 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bc74a93-81cf-43ce-8056-a60d205aeb6c None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:56:42.627959 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-8bc74a93-81cf-43ce-8056-a60d205aeb6c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:56:42.665398 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:56:42.733124 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bc74a93-81cf-43ce-8056-a60d205aeb6c None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:56:42.805615 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f19bb698-7a3b-48cb-b482-13f6625525ff service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f3ea1580c56f4814a92bbb2acf542905', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4562f3951fbe465280073e60a11e44e2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:56:42.806216 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f19bb698-7a3b-48cb-b482-13f6625525ff service neutron] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:56:42.806496 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f19bb698-7a3b-48cb-b482-13f6625525ff service neutron] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:56:42.806766 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f19bb698-7a3b-48cb-b482-13f6625525ff service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:56:42.808356 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-f19bb698-7a3b-48cb-b482-13f6625525ff service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:56:42.888106 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:56:42.916875 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f19bb698-7a3b-48cb-b482-13f6625525ff service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:56:42.918339 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f19bb698-7a3b-48cb-b482-13f6625525ff service neutron] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:56:42.936291 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 751/1490] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Aug 30 01:56:42 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 310 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 01:56:45.317928 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edbccb1b-ee97-4828-9c3e-af09d0948488 None None] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:56:45.318544 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edbccb1b-ee97-4828-9c3e-af09d0948488 None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:56:45.318544 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edbccb1b-ee97-4828-9c3e-af09d0948488 None None] PATH_INFO: `/` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:56:45.320503 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 740/1491] 10.222.0.27 () {54 vars in 927 bytes} [Tue Aug 30 01:56:45 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 01:56:45.326014 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b974fc9-7e0f-4e9a-bcf3-d4ef3647b86e None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:56:45.326447 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b974fc9-7e0f-4e9a-bcf3-d4ef3647b86e None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:56:45.326591 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b974fc9-7e0f-4e9a-bcf3-d4ef3647b86e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:56:45.392203 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-9b974fc9-7e0f-4e9a-bcf3-d4ef3647b86e None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 01:56:45.393424 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-9b974fc9-7e0f-4e9a-bcf3-d4ef3647b86e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:56:45.489237 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 752/1492] 10.222.0.27 () {58 vars in 1024 bytes} [Tue Aug 30 01:56:45 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 164 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 01:56:45.783483 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edbccb1b-ee97-4828-9c3e-af09d0948488 None None] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:56:45.785093 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-edbccb1b-ee97-4828-9c3e-af09d0948488 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:56:45.887030 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edbccb1b-ee97-4828-9c3e-af09d0948488 None None] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:56:45.964223 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95df2a1c-7dda-44bb-a7cc-e1fed655b245 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f3ea1580c56f4814a92bbb2acf542905', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4562f3951fbe465280073e60a11e44e2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:56:45.964830 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95df2a1c-7dda-44bb-a7cc-e1fed655b245 service neutron] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:56:45.965112 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95df2a1c-7dda-44bb-a7cc-e1fed655b245 service neutron] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:56:45.965380 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95df2a1c-7dda-44bb-a7cc-e1fed655b245 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:56:45.967016 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-95df2a1c-7dda-44bb-a7cc-e1fed655b245 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:56:45.976634 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:56:46.074888 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95df2a1c-7dda-44bb-a7cc-e1fed655b245 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:56:46.076324 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95df2a1c-7dda-44bb-a7cc-e1fed655b245 service neutron] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:56:46.093996 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 741/1493] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Aug 30 01:56:45 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 311 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 01:56:55.487469 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=82 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:56:56.091692 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=50 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:58:31.101928 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7858274d-4913-42bf-87cc-9a812bb2d7e1 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:58:31.102529 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7858274d-4913-42bf-87cc-9a812bb2d7e1 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:58:31.102529 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7858274d-4913-42bf-87cc-9a812bb2d7e1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:58:31.182432 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-7858274d-4913-42bf-87cc-9a812bb2d7e1 None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 01:58:31.183664 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-7858274d-4913-42bf-87cc-9a812bb2d7e1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:58:31.280191 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 753/1494] 10.222.0.27 () {58 vars in 958 bytes} [Tue Aug 30 01:58:31 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 179 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 01:58:31.288432 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7c346d2-d02f-484f-bc01-be42071f1d2e None None] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:58:31.288937 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7c346d2-d02f-484f-bc01-be42071f1d2e None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:58:31.289050 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7c346d2-d02f-484f-bc01-be42071f1d2e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:58:31.364251 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.auth.core [None req-f7c346d2-d02f-484f-bc01-be42071f1d2e None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81578) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 01:58:31.365357 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-f7c346d2-d02f-484f-bc01-be42071f1d2e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:58:31.460691 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 742/1495] 10.222.0.27 () {58 vars in 958 bytes} [Tue Aug 30 01:58:31 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 173 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 01:58:31.465279 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7858274d-4913-42bf-87cc-9a812bb2d7e1 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:58:31.466495 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-7858274d-4913-42bf-87cc-9a812bb2d7e1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:58:31.478167 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7858274d-4913-42bf-87cc-9a812bb2d7e1 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:58:31.538022 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:58:31.571427 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb1d7390-da2d-4a47-8bc6-316c1724a3b6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:58:31.572033 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb1d7390-da2d-4a47-8bc6-316c1724a3b6 admin admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:58:31.572310 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb1d7390-da2d-4a47-8bc6-316c1724a3b6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:58:31.572576 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb1d7390-da2d-4a47-8bc6-316c1724a3b6 admin admin] PATH_INFO: `/v3/domains` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:58:31.573218 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb1d7390-da2d-4a47-8bc6-316c1724a3b6 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 01:58:31.573561 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb1d7390-da2d-4a47-8bc6-316c1724a3b6 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:58:31.576020 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: WARNING py.warnings [None req-eb1d7390-da2d-4a47-8bc6-316c1724a3b6 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:list_domains": "role:reader and system_scope:all" failed scope check. The token used to make the request was project scoped but the policy requires ['system'] scope. This behavior may change in the future where using the intended scope is required Aug 30 01:58:31.576020 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: warnings.warn(msg) Aug 30 01:58:31.576020 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: Aug 30 01:58:31.576283 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb1d7390-da2d-4a47-8bc6-316c1724a3b6 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:58:31.587967 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 754/1496] 10.222.0.27 () {58 vars in 1153 bytes} [Tue Aug 30 01:58:31 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 123 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 01:58:31.596470 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7c346d2-d02f-484f-bc01-be42071f1d2e None None] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:58:31.605614 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7c346d2-d02f-484f-bc01-be42071f1d2e None None] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:58:31.668070 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:58:31.702188 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bca3c97-8b12-4799-a9e7-1bb7bda39aff admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:58:31.702802 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bca3c97-8b12-4799-a9e7-1bb7bda39aff admin admin] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:58:31.703080 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bca3c97-8b12-4799-a9e7-1bb7bda39aff admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:58:31.703348 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bca3c97-8b12-4799-a9e7-1bb7bda39aff admin admin] PATH_INFO: `/v3/projects` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:58:31.703970 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4bca3c97-8b12-4799-a9e7-1bb7bda39aff admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:58:31.705803 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: WARNING py.warnings [None req-4bca3c97-8b12-4799-a9e7-1bb7bda39aff admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_project": "(role:admin and system_scope:all) or (role:admin and domain_id:%(target.project.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Aug 30 01:58:31.705803 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: warnings.warn(msg) Aug 30 01:58:31.705803 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: Aug 30 01:58:31.707037 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4bca3c97-8b12-4799-a9e7-1bb7bda39aff admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:58:31.874212 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 743/1497] 10.222.0.27 () {60 vars in 1153 bytes} [Tue Aug 30 01:58:31 2022] POST /identity/v3/projects => generated 377 bytes in 278 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 30 01:58:31.878632 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb1d7390-da2d-4a47-8bc6-316c1724a3b6 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:58:31.880132 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-eb1d7390-da2d-4a47-8bc6-316c1724a3b6 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:58:31.913345 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb1d7390-da2d-4a47-8bc6-316c1724a3b6 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:58:32.009973 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df98e3a9-ee9a-44a0-9f47-ad8b52d17dcf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:58:32.010488 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df98e3a9-ee9a-44a0-9f47-ad8b52d17dcf admin admin] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:58:32.010761 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df98e3a9-ee9a-44a0-9f47-ad8b52d17dcf admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:58:32.011024 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df98e3a9-ee9a-44a0-9f47-ad8b52d17dcf admin admin] PATH_INFO: `/v3/users` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:58:32.011645 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df98e3a9-ee9a-44a0-9f47-ad8b52d17dcf admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:58:32.012650 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: WARNING py.warnings [None req-df98e3a9-ee9a-44a0-9f47-ad8b52d17dcf admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_user": "(role:admin and system_scope:all) or (role:admin and token.domain.id:%(target.user.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Aug 30 01:58:32.012650 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: warnings.warn(msg) Aug 30 01:58:32.012650 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: Aug 30 01:58:32.013854 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df98e3a9-ee9a-44a0-9f47-ad8b52d17dcf admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:58:32.157908 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 755/1498] 10.222.0.27 () {60 vars in 1144 bytes} [Tue Aug 30 01:58:31 2022] POST /identity/v3/users => generated 289 bytes in 280 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 01:58:32.162625 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bca3c97-8b12-4799-a9e7-1bb7bda39aff admin admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:58:32.164124 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-4bca3c97-8b12-4799-a9e7-1bb7bda39aff admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:58:32.225797 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bca3c97-8b12-4799-a9e7-1bb7bda39aff admin admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:58:32.308424 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:58:32.318666 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-451ef3aa-4dd4-4abb-873b-48294a586d22 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:58:32.319261 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-451ef3aa-4dd4-4abb-873b-48294a586d22 admin admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:58:32.319541 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-451ef3aa-4dd4-4abb-873b-48294a586d22 admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:58:32.326406 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-451ef3aa-4dd4-4abb-873b-48294a586d22 admin admin] PATH_INFO: `/v3/roles` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:58:32.326406 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-451ef3aa-4dd4-4abb-873b-48294a586d22 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:58:32.326406 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: WARNING py.warnings [None req-451ef3aa-4dd4-4abb-873b-48294a586d22 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:list_roles": "role:reader and system_scope:all" failed scope check. The token used to make the request was project scoped but the policy requires ['system'] scope. This behavior may change in the future where using the intended scope is required Aug 30 01:58:32.326406 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: warnings.warn(msg) Aug 30 01:58:32.326406 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: Aug 30 01:58:32.326406 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-451ef3aa-4dd4-4abb-873b-48294a586d22 admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:58:32.334957 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 744/1499] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Aug 30 01:58:32 2022] GET /identity/v3/roles => generated 1420 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 30 01:58:32.339021 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df98e3a9-ee9a-44a0-9f47-ad8b52d17dcf admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:58:32.340577 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-df98e3a9-ee9a-44a0-9f47-ad8b52d17dcf admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:58:32.353072 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df98e3a9-ee9a-44a0-9f47-ad8b52d17dcf admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:58:32.378185 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:58:32.445643 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7aebd1f6-f744-4cc5-afe9-00607cd970a3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:58:32.446247 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7aebd1f6-f744-4cc5-afe9-00607cd970a3 admin admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:58:32.446522 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7aebd1f6-f744-4cc5-afe9-00607cd970a3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:58:32.446793 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7aebd1f6-f744-4cc5-afe9-00607cd970a3 admin admin] PATH_INFO: `/v3/roles` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:58:32.447425 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7aebd1f6-f744-4cc5-afe9-00607cd970a3 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:58:32.448431 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: WARNING py.warnings [None req-7aebd1f6-f744-4cc5-afe9-00607cd970a3 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:list_roles": "role:reader and system_scope:all" failed scope check. The token used to make the request was project scoped but the policy requires ['system'] scope. This behavior may change in the future where using the intended scope is required Aug 30 01:58:32.448431 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: warnings.warn(msg) Aug 30 01:58:32.448431 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: Aug 30 01:58:32.449306 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7aebd1f6-f744-4cc5-afe9-00607cd970a3 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:58:32.462220 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 756/1500] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Aug 30 01:58:32 2022] GET /identity/v3/roles => generated 1420 bytes in 124 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 01:58:32.466428 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-451ef3aa-4dd4-4abb-873b-48294a586d22 admin admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:58:32.467947 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-451ef3aa-4dd4-4abb-873b-48294a586d22 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:58:32.501724 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-451ef3aa-4dd4-4abb-873b-48294a586d22 admin admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:58:32.592957 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd0f9f11-da0b-4e0f-8b6d-d4ab7bad34b7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:58:32.593558 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd0f9f11-da0b-4e0f-8b6d-d4ab7bad34b7 admin admin] REQUEST_METHOD: `PUT` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:58:32.593836 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd0f9f11-da0b-4e0f-8b6d-d4ab7bad34b7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:58:32.594102 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd0f9f11-da0b-4e0f-8b6d-d4ab7bad34b7 admin admin] PATH_INFO: `/v3/projects/17b44539143d4ead9274b1d5b4ffeae2/users/cda030c28f53490dad014dbfcae53193/roles/1fb81afcb9f245f3b989add94598a063` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:58:32.616368 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd0f9f11-da0b-4e0f-8b6d-d4ab7bad34b7 admin admin] RBAC: Authorizing `identity:create_grant(project_id=17b44539143d4ead9274b1d5b4ffeae2, user_id=cda030c28f53490dad014dbfcae53193, role_id=1fb81afcb9f245f3b989add94598a063)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:58:32.617377 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: WARNING py.warnings [None req-fd0f9f11-da0b-4e0f-8b6d-d4ab7bad34b7 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_grant": "(role:admin and system_scope:all) or ((role:admin and domain_id:%(target.user.domain_id)s and domain_id:%(target.project.domain_id)s) or (role:admin and domain_id:%(target.user.domain_id)s and domain_id:%(target.domain.id)s) or (role:admin and domain_id:%(target.group.domain_id)s and domain_id:%(target.project.domain_id)s) or (role:admin and domain_id:%(target.group.domain_id)s and domain_id:%(target.domain.id)s)) and (domain_id:%(target.role.domain_id)s or None:%(target.role.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Aug 30 01:58:32.617377 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: warnings.warn(msg) Aug 30 01:58:32.617377 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: Aug 30 01:58:32.619329 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd0f9f11-da0b-4e0f-8b6d-d4ab7bad34b7 admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:58:32.741184 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 745/1501] 10.222.0.27 () {60 vars in 1483 bytes} [Tue Aug 30 01:58:32 2022] PUT /identity/v3/projects/17b44539143d4ead9274b1d5b4ffeae2/users/cda030c28f53490dad014dbfcae53193/roles/1fb81afcb9f245f3b989add94598a063 => generated 0 bytes in 275 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 01:58:32.754049 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-825c9ff9-d892-4ea3-a155-044b464885d7 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:58:32.754350 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-825c9ff9-d892-4ea3-a155-044b464885d7 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:58:32.754621 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-825c9ff9-d892-4ea3-a155-044b464885d7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:58:32.811268 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-825c9ff9-d892-4ea3-a155-044b464885d7 None None] MFA Rules not processed for user `cda030c28f53490dad014dbfcae53193`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 01:58:32.812476 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-825c9ff9-d892-4ea3-a155-044b464885d7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:58:32.888463 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 757/1502] 10.222.0.27 () {58 vars in 958 bytes} [Tue Aug 30 01:58:32 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 136 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 30 01:58:32.902039 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e10e55c2-7e34-485c-8b05-859965261845 None None] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:58:32.902325 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e10e55c2-7e34-485c-8b05-859965261845 None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:58:32.902595 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e10e55c2-7e34-485c-8b05-859965261845 None None] PATH_INFO: `/` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:58:32.903948 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 746/1503] 10.222.0.27 () {54 vars in 964 bytes} [Tue Aug 30 01:58:32 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 01:58:32.908716 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c88e792-e391-47a8-9b9d-7cd4ed864590 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:58:32.909006 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c88e792-e391-47a8-9b9d-7cd4ed864590 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:58:32.909278 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c88e792-e391-47a8-9b9d-7cd4ed864590 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:58:32.939625 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:58:32.984781 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-7c88e792-e391-47a8-9b9d-7cd4ed864590 None None] MFA Rules not processed for user `ab936b3767a8417aa80c8c8f1b84b667`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 01:58:32.986025 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-7c88e792-e391-47a8-9b9d-7cd4ed864590 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:58:33.062307 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 758/1504] 10.222.0.27 () {58 vars in 1061 bytes} [Tue Aug 30 01:58:32 2022] POST /identity/v3/auth/tokens => generated 3120 bytes in 154 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 01:58:33.070697 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e10e55c2-7e34-485c-8b05-859965261845 None None] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:58:33.074873 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-e10e55c2-7e34-485c-8b05-859965261845 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:58:33.129352 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:58:33.185886 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e10e55c2-7e34-485c-8b05-859965261845 None None] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:58:33.189534 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47dfa067-c43a-4af4-bf9a-2dcf4d07b5db service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ab936b3767a8417aa80c8c8f1b84b667', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4562f3951fbe465280073e60a11e44e2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:58:33.190745 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47dfa067-c43a-4af4-bf9a-2dcf4d07b5db service cinder] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:58:33.191027 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47dfa067-c43a-4af4-bf9a-2dcf4d07b5db service cinder] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:58:33.191297 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47dfa067-c43a-4af4-bf9a-2dcf4d07b5db service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:58:33.193067 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-47dfa067-c43a-4af4-bf9a-2dcf4d07b5db service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:58:33.290650 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47dfa067-c43a-4af4-bf9a-2dcf4d07b5db service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:58:33.292143 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47dfa067-c43a-4af4-bf9a-2dcf4d07b5db service cinder] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:58:33.311136 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 747/1505] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Aug 30 01:58:33 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 241 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 01:58:33.362370 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41263806-88af-4661-9bb3-e427c7705d62 None None] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:58:33.363027 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41263806-88af-4661-9bb3-e427c7705d62 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:58:33.363027 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41263806-88af-4661-9bb3-e427c7705d62 None None] PATH_INFO: `/` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:58:33.364222 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 759/1506] 10.222.0.27 () {52 vars in 830 bytes} [Tue Aug 30 01:58:33 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 01:58:33.367970 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47dfa067-c43a-4af4-bf9a-2dcf4d07b5db service cinder] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:58:33.369396 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-47dfa067-c43a-4af4-bf9a-2dcf4d07b5db service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:58:33.444820 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:58:33.474297 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47dfa067-c43a-4af4-bf9a-2dcf4d07b5db service cinder] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:58:33.479089 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddb3cea1-55cd-4746-b8ec-06d7b3ba698d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:58:33.479750 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddb3cea1-55cd-4746-b8ec-06d7b3ba698d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:58:33.480027 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddb3cea1-55cd-4746-b8ec-06d7b3ba698d admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:58:33.480293 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddb3cea1-55cd-4746-b8ec-06d7b3ba698d admin admin] PATH_INFO: `/v3/users/cda030c28f53490dad014dbfcae53193` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:58:33.481406 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ddb3cea1-55cd-4746-b8ec-06d7b3ba698d admin admin] RBAC: Authorizing `identity:delete_user(user_id=cda030c28f53490dad014dbfcae53193)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:58:33.482440 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: WARNING py.warnings [None req-ddb3cea1-55cd-4746-b8ec-06d7b3ba698d admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:delete_user": "(role:admin and system_scope:all) or (role:admin and token.domain.id:%(target.user.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Aug 30 01:58:33.482440 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: warnings.warn(msg) Aug 30 01:58:33.482440 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: Aug 30 01:58:33.483627 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ddb3cea1-55cd-4746-b8ec-06d7b3ba698d admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:58:33.817838 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-ddb3cea1-55cd-4746-b8ec-06d7b3ba698d admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'cda030c28f53490dad014dbfcae53193'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 01:58:33.818598 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-ddb3cea1-55cd-4746-b8ec-06d7b3ba698d admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'cda030c28f53490dad014dbfcae53193'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 01:58:33.940779 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-ddb3cea1-55cd-4746-b8ec-06d7b3ba698d admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'cda030c28f53490dad014dbfcae53193'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 01:58:33.941414 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-ddb3cea1-55cd-4746-b8ec-06d7b3ba698d admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'cda030c28f53490dad014dbfcae53193'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 01:58:33.956116 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-ddb3cea1-55cd-4746-b8ec-06d7b3ba698d admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'cda030c28f53490dad014dbfcae53193'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 01:58:33.991114 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 748/1507] 10.222.0.27 () {58 vars in 1224 bytes} [Tue Aug 30 01:58:33 2022] DELETE /identity/v3/users/cda030c28f53490dad014dbfcae53193 => generated 0 bytes in 624 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 01:58:33.995911 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9add252-4b74-43ee-a46a-94edc91d7660 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:58:33.996194 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9add252-4b74-43ee-a46a-94edc91d7660 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:58:33.996468 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9add252-4b74-43ee-a46a-94edc91d7660 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:58:34.029953 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-c9add252-4b74-43ee-a46a-94edc91d7660 None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 01:58:34.031247 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-c9add252-4b74-43ee-a46a-94edc91d7660 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:58:34.090675 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 760/1508] 10.222.0.27 () {58 vars in 958 bytes} [Tue Aug 30 01:58:33 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 01:58:34.100999 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddb3cea1-55cd-4746-b8ec-06d7b3ba698d admin admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:58:34.102424 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-ddb3cea1-55cd-4746-b8ec-06d7b3ba698d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:58:34.181299 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddb3cea1-55cd-4746-b8ec-06d7b3ba698d admin admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:58:34.184997 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9b98a3e-eab8-4530-a64e-f6fe363e59bc service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f3ea1580c56f4814a92bbb2acf542905', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4562f3951fbe465280073e60a11e44e2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:58:34.185633 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9b98a3e-eab8-4530-a64e-f6fe363e59bc service neutron] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:58:34.185910 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9b98a3e-eab8-4530-a64e-f6fe363e59bc service neutron] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:58:34.186177 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9b98a3e-eab8-4530-a64e-f6fe363e59bc service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:58:34.195277 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9b98a3e-eab8-4530-a64e-f6fe363e59bc service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:58:34.196720 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9b98a3e-eab8-4530-a64e-f6fe363e59bc service neutron] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:58:34.206199 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 749/1509] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Aug 30 01:58:34 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 106 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 01:58:34.852040 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9add252-4b74-43ee-a46a-94edc91d7660 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 01:58:34.853520 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-c9add252-4b74-43ee-a46a-94edc91d7660 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 01:58:34.866700 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9add252-4b74-43ee-a46a-94edc91d7660 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 01:58:34.872310 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-577d59c7-9d47-42e4-ab10-05a40a646f86 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 01:58:34.872943 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-577d59c7-9d47-42e4-ab10-05a40a646f86 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 01:58:34.873216 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-577d59c7-9d47-42e4-ab10-05a40a646f86 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 01:58:34.873573 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-577d59c7-9d47-42e4-ab10-05a40a646f86 admin admin] PATH_INFO: `/v3/projects/17b44539143d4ead9274b1d5b4ffeae2` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 01:58:34.874703 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-577d59c7-9d47-42e4-ab10-05a40a646f86 admin admin] RBAC: Authorizing `identity:delete_project(project_id=17b44539143d4ead9274b1d5b4ffeae2)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 01:58:34.875715 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: WARNING py.warnings [None req-577d59c7-9d47-42e4-ab10-05a40a646f86 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:delete_project": "(role:admin and system_scope:all) or (role:admin and domain_id:%(target.project.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Aug 30 01:58:34.875715 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: warnings.warn(msg) Aug 30 01:58:34.875715 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: Aug 30 01:58:34.876898 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-577d59c7-9d47-42e4-ab10-05a40a646f86 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 01:58:35.092100 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=97,keystone:DELETE=3 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:58:35.095593 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-577d59c7-9d47-42e4-ab10-05a40a646f86 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '17b44539143d4ead9274b1d5b4ffeae2', 'request_id': 'req-577d59c7-9d47-42e4-ab10-05a40a646f86'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 01:58:35.101749 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-577d59c7-9d47-42e4-ab10-05a40a646f86 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '17b44539143d4ead9274b1d5b4ffeae2', 'request_id': 'req-577d59c7-9d47-42e4-ab10-05a40a646f86'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 01:58:35.115183 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-577d59c7-9d47-42e4-ab10-05a40a646f86 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '17b44539143d4ead9274b1d5b4ffeae2', 'request_id': 'req-577d59c7-9d47-42e4-ab10-05a40a646f86'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 01:58:35.328223 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-577d59c7-9d47-42e4-ab10-05a40a646f86 admin admin] The token cache is being invalidate because project 17b44539143d4ead9274b1d5b4ffeae2 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81577) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 30 01:58:35.328512 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-577d59c7-9d47-42e4-ab10-05a40a646f86 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 01:58:35.329136 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-577d59c7-9d47-42e4-ab10-05a40a646f86 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 01:58:35.330840 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 761/1510] 10.222.0.27 () {58 vars in 1233 bytes} [Tue Aug 30 01:58:34 2022] DELETE /identity/v3/projects/17b44539143d4ead9274b1d5b4ffeae2 => generated 0 bytes in 479 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 01:58:44.201921 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=72,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 01:58:45.369175 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:DELETE=4,keystone:SELECT=7,keystone:INSERT=1 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:00:01.884695 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90dd06bb-2658-412f-954c-8ee107c4c697 None None] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:00:01.885339 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90dd06bb-2658-412f-954c-8ee107c4c697 None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:00:01.885339 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90dd06bb-2658-412f-954c-8ee107c4c697 None None] PATH_INFO: `/` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:00:01.886591 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 750/1511] 127.0.0.1 () {54 vars in 901 bytes} [Tue Aug 30 02:00:01 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 02:00:01.892030 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90213618-a045-43ab-95ab-931283beb5f8 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:00:01.892529 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90213618-a045-43ab-95ab-931283beb5f8 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:00:01.892634 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90213618-a045-43ab-95ab-931283beb5f8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:00:01.942379 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-90213618-a045-43ab-95ab-931283beb5f8 None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:00:01.943579 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-90213618-a045-43ab-95ab-931283beb5f8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:00:02.002450 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 762/1512] 127.0.0.1 () {58 vars in 998 bytes} [Tue Aug 30 02:00:01 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 111 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 30 02:00:02.157143 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90dd06bb-2658-412f-954c-8ee107c4c697 None None] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:00:02.158885 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-90dd06bb-2658-412f-954c-8ee107c4c697 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:00:02.261826 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90dd06bb-2658-412f-954c-8ee107c4c697 None None] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:00:02.267807 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49fff5cf-98cf-4c5e-b7a1-a54ef9ab7fdd service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd64f2e71361644589bec8a06e6eceed1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4562f3951fbe465280073e60a11e44e2', 'project_domain_id': 'default', 'roles': ['member', 'service', 'admin', 'ResellerAdmin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:00:02.268403 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49fff5cf-98cf-4c5e-b7a1-a54ef9ab7fdd service nova] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:00:02.268678 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49fff5cf-98cf-4c5e-b7a1-a54ef9ab7fdd service nova] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:00:02.268945 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49fff5cf-98cf-4c5e-b7a1-a54ef9ab7fdd service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:00:02.276732 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49fff5cf-98cf-4c5e-b7a1-a54ef9ab7fdd service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:00:02.278439 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49fff5cf-98cf-4c5e-b7a1-a54ef9ab7fdd service nova] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:00:02.287661 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 751/1513] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Aug 30 02:00:02 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 131 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 02:00:03.138331 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2741f0c-dbe2-445b-883b-912b6ea9f169 None None] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:00:03.138878 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2741f0c-dbe2-445b-883b-912b6ea9f169 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:00:03.138999 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2741f0c-dbe2-445b-883b-912b6ea9f169 None None] PATH_INFO: `/` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:00:03.140350 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 763/1514] 127.0.0.1 () {54 vars in 901 bytes} [Tue Aug 30 02:00:03 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 02:00:03.145252 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7990ce2-fab4-431d-bce3-874f393beadd None None] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:00:03.145681 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7990ce2-fab4-431d-bce3-874f393beadd None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:00:03.145829 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7990ce2-fab4-431d-bce3-874f393beadd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:00:03.165678 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.auth.core [None req-e7990ce2-fab4-431d-bce3-874f393beadd None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81578) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:00:03.166901 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-e7990ce2-fab4-431d-bce3-874f393beadd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:00:03.175267 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 752/1515] 127.0.0.1 () {58 vars in 998 bytes} [Tue Aug 30 02:00:03 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 02:00:03.329172 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2741f0c-dbe2-445b-883b-912b6ea9f169 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:00:03.344754 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2741f0c-dbe2-445b-883b-912b6ea9f169 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:00:03.350030 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c7d3132-c991-4bdc-aa2b-e37d63041fb7 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd64f2e71361644589bec8a06e6eceed1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4562f3951fbe465280073e60a11e44e2', 'project_domain_id': 'default', 'roles': ['member', 'service', 'admin', 'ResellerAdmin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:00:03.350618 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c7d3132-c991-4bdc-aa2b-e37d63041fb7 service nova] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:00:03.350892 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c7d3132-c991-4bdc-aa2b-e37d63041fb7 service nova] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:00:03.351158 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c7d3132-c991-4bdc-aa2b-e37d63041fb7 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:00:03.359389 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c7d3132-c991-4bdc-aa2b-e37d63041fb7 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:00:03.360997 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c7d3132-c991-4bdc-aa2b-e37d63041fb7 service nova] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:00:03.370128 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 764/1516] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Aug 30 02:00:03 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 41 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 02:00:04.108611 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2035cd7b-b5bf-4671-8f6d-f2d6296013b6 None None] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:00:04.108890 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2035cd7b-b5bf-4671-8f6d-f2d6296013b6 None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:00:04.109164 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2035cd7b-b5bf-4671-8f6d-f2d6296013b6 None None] PATH_INFO: `/v3` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:00:04.110476 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 753/1517] 10.222.0.42 () {54 vars in 925 bytes} [Tue Aug 30 02:00:04 2022] GET /identity/v3 => generated 255 bytes in 3 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 02:00:04.116502 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91785d7e-9b1f-4ea2-bb1d-d4f59220987d None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:00:04.116783 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91785d7e-9b1f-4ea2-bb1d-d4f59220987d None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:00:04.117086 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91785d7e-9b1f-4ea2-bb1d-d4f59220987d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:00:04.177613 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-91785d7e-9b1f-4ea2-bb1d-d4f59220987d None None] MFA Rules not processed for user `379536ff06fd4f719899e5dde732e63d`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:00:04.178837 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-91785d7e-9b1f-4ea2-bb1d-d4f59220987d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:00:04.250935 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 765/1518] 10.222.0.42 () {58 vars in 1015 bytes} [Tue Aug 30 02:00:04 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 135 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 02:00:08.321557 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-201744ca-0308-4ac3-a8fc-79efae678645 None None] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:00:08.322099 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-201744ca-0308-4ac3-a8fc-79efae678645 None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:00:08.322099 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-201744ca-0308-4ac3-a8fc-79efae678645 None None] PATH_INFO: `/` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:00:08.323379 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 754/1519] 10.222.0.27 () {54 vars in 961 bytes} [Tue Aug 30 02:00:08 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 02:00:08.328655 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd9fb956-8ffc-423c-aab2-be48c72d0ff9 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:00:08.329114 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd9fb956-8ffc-423c-aab2-be48c72d0ff9 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:00:08.329214 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd9fb956-8ffc-423c-aab2-be48c72d0ff9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:00:08.349664 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-fd9fb956-8ffc-423c-aab2-be48c72d0ff9 None None] MFA Rules not processed for user `379536ff06fd4f719899e5dde732e63d`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:00:08.350861 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-fd9fb956-8ffc-423c-aab2-be48c72d0ff9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:00:08.360011 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 766/1520] 10.222.0.27 () {58 vars in 1058 bytes} [Tue Aug 30 02:00:08 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 30 02:00:08.367725 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-201744ca-0308-4ac3-a8fc-79efae678645 None None] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:00:08.380861 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-201744ca-0308-4ac3-a8fc-79efae678645 None None] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:00:08.385914 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dec1177c-a815-4b6d-a4df-55fd601b0fcd service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '379536ff06fd4f719899e5dde732e63d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4562f3951fbe465280073e60a11e44e2', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:00:08.386503 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dec1177c-a815-4b6d-a4df-55fd601b0fcd service placement] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:00:08.386783 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dec1177c-a815-4b6d-a4df-55fd601b0fcd service placement] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:00:08.387055 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dec1177c-a815-4b6d-a4df-55fd601b0fcd service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:00:08.397972 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dec1177c-a815-4b6d-a4df-55fd601b0fcd service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:00:08.399609 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dec1177c-a815-4b6d-a4df-55fd601b0fcd service placement] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:00:08.410184 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 755/1521] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Aug 30 02:00:08 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 43 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 02:00:14.573028 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72b9d25a-75e4-4549-b0a1-261e2706a972 None None] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:00:14.573612 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72b9d25a-75e4-4549-b0a1-261e2706a972 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:00:14.573612 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72b9d25a-75e4-4549-b0a1-261e2706a972 None None] PATH_INFO: `/` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:00:14.575101 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 767/1522] 127.0.0.1 () {54 vars in 901 bytes} [Tue Aug 30 02:00:14 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 02:00:14.580016 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edbc4223-933e-4ae4-ba43-ec04af6b2085 None None] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:00:14.580425 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edbc4223-933e-4ae4-ba43-ec04af6b2085 None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:00:14.580592 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edbc4223-933e-4ae4-ba43-ec04af6b2085 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:00:14.601570 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.auth.core [None req-edbc4223-933e-4ae4-ba43-ec04af6b2085 None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81578) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:00:14.602764 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-edbc4223-933e-4ae4-ba43-ec04af6b2085 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:00:14.611538 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 756/1523] 127.0.0.1 () {58 vars in 998 bytes} [Tue Aug 30 02:00:14 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 02:00:14.769113 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72b9d25a-75e4-4549-b0a1-261e2706a972 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:00:14.783579 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72b9d25a-75e4-4549-b0a1-261e2706a972 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:00:14.789097 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bdbc449-ac8b-4cf1-9e3d-a798179fe827 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd64f2e71361644589bec8a06e6eceed1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4562f3951fbe465280073e60a11e44e2', 'project_domain_id': 'default', 'roles': ['member', 'service', 'admin', 'ResellerAdmin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:00:14.789718 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bdbc449-ac8b-4cf1-9e3d-a798179fe827 service nova] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:00:14.789995 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bdbc449-ac8b-4cf1-9e3d-a798179fe827 service nova] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:00:14.790261 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bdbc449-ac8b-4cf1-9e3d-a798179fe827 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:00:14.799069 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5bdbc449-ac8b-4cf1-9e3d-a798179fe827 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:00:14.800630 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5bdbc449-ac8b-4cf1-9e3d-a798179fe827 service nova] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:00:14.804568 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:00:14.810584 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 768/1524] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Aug 30 02:00:14 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 42 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 02:00:15.631091 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8136b8d0-97ab-4196-bf35-594136098e7f None None] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:00:15.631678 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8136b8d0-97ab-4196-bf35-594136098e7f None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:00:15.631678 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8136b8d0-97ab-4196-bf35-594136098e7f None None] PATH_INFO: `/` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:00:15.632986 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 757/1525] 127.0.0.1 () {54 vars in 901 bytes} [Tue Aug 30 02:00:15 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 02:00:15.637782 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb1e84ae-cdfb-4093-8a4b-222f3bba091b None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:00:15.638210 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb1e84ae-cdfb-4093-8a4b-222f3bba091b None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:00:15.638322 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb1e84ae-cdfb-4093-8a4b-222f3bba091b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:00:15.658528 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-bb1e84ae-cdfb-4093-8a4b-222f3bba091b None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:00:15.659733 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-bb1e84ae-cdfb-4093-8a4b-222f3bba091b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:00:15.667474 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 769/1526] 127.0.0.1 () {58 vars in 998 bytes} [Tue Aug 30 02:00:15 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 02:00:15.821039 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8136b8d0-97ab-4196-bf35-594136098e7f None None] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:00:15.835557 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8136b8d0-97ab-4196-bf35-594136098e7f None None] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:00:15.840861 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f54c5236-723a-4ca1-88b2-76d0b0a29a8c service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd64f2e71361644589bec8a06e6eceed1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4562f3951fbe465280073e60a11e44e2', 'project_domain_id': 'default', 'roles': ['member', 'service', 'admin', 'ResellerAdmin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:00:15.841444 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f54c5236-723a-4ca1-88b2-76d0b0a29a8c service nova] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:00:15.841745 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f54c5236-723a-4ca1-88b2-76d0b0a29a8c service nova] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:00:15.842012 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f54c5236-723a-4ca1-88b2-76d0b0a29a8c service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:00:15.850179 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f54c5236-723a-4ca1-88b2-76d0b0a29a8c service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:00:15.851734 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f54c5236-723a-4ca1-88b2-76d0b0a29a8c service nova] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:00:15.860854 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 758/1527] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Aug 30 02:00:15 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 40 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 02:00:19.551603 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b35e623d-c803-4654-a785-698814c7fe76 None None] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:00:19.552145 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b35e623d-c803-4654-a785-698814c7fe76 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:00:19.552145 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b35e623d-c803-4654-a785-698814c7fe76 None None] PATH_INFO: `/` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:00:19.553414 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 770/1528] 127.0.0.1 () {54 vars in 917 bytes} [Tue Aug 30 02:00:19 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 02:00:19.558803 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-292959b3-41ab-497c-8a9d-fd5c10f52cb7 None None] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:00:19.563705 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-292959b3-41ab-497c-8a9d-fd5c10f52cb7 None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:00:19.563705 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-292959b3-41ab-497c-8a9d-fd5c10f52cb7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:00:19.580868 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.auth.core [None req-292959b3-41ab-497c-8a9d-fd5c10f52cb7 None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81578) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:00:19.582149 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-292959b3-41ab-497c-8a9d-fd5c10f52cb7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:00:19.589773 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 759/1529] 127.0.0.1 () {58 vars in 1014 bytes} [Tue Aug 30 02:00:19 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 30 02:00:19.881425 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b35e623d-c803-4654-a785-698814c7fe76 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:00:19.883146 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-b35e623d-c803-4654-a785-698814c7fe76 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:00:19.940285 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b35e623d-c803-4654-a785-698814c7fe76 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:00:19.943805 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3d5def0-af20-4f17-9248-2aa1cbb06c4c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f3ea1580c56f4814a92bbb2acf542905', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4562f3951fbe465280073e60a11e44e2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:00:19.944483 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3d5def0-af20-4f17-9248-2aa1cbb06c4c service neutron] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:00:19.944759 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3d5def0-af20-4f17-9248-2aa1cbb06c4c service neutron] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:00:19.945026 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3d5def0-af20-4f17-9248-2aa1cbb06c4c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:00:19.954532 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3d5def0-af20-4f17-9248-2aa1cbb06c4c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:00:19.955949 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3d5def0-af20-4f17-9248-2aa1cbb06c4c service neutron] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:00:19.965424 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 771/1530] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Aug 30 02:00:19 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 02:00:21.916004 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d81012d-200d-49a1-b988-4f7f9b7eb3e9 None None] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:00:21.916588 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d81012d-200d-49a1-b988-4f7f9b7eb3e9 None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:00:21.916588 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d81012d-200d-49a1-b988-4f7f9b7eb3e9 None None] PATH_INFO: `/` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:00:21.917926 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 760/1531] 127.0.0.1 () {54 vars in 917 bytes} [Tue Aug 30 02:00:21 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 02:00:21.923256 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a34bbbc5-d914-4949-babc-66b1133b4d8d None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:00:21.923667 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a34bbbc5-d914-4949-babc-66b1133b4d8d None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:00:21.923803 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a34bbbc5-d914-4949-babc-66b1133b4d8d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:00:21.943914 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-a34bbbc5-d914-4949-babc-66b1133b4d8d None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:00:21.945118 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-a34bbbc5-d914-4949-babc-66b1133b4d8d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:00:21.953713 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 772/1532] 127.0.0.1 () {58 vars in 1014 bytes} [Tue Aug 30 02:00:21 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 02:00:22.241572 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d81012d-200d-49a1-b988-4f7f9b7eb3e9 None None] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:00:22.252944 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d81012d-200d-49a1-b988-4f7f9b7eb3e9 None None] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:00:22.256631 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07244365-3feb-4c2c-a1bf-9c4b62eeff52 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f3ea1580c56f4814a92bbb2acf542905', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4562f3951fbe465280073e60a11e44e2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:00:22.257232 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07244365-3feb-4c2c-a1bf-9c4b62eeff52 service neutron] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:00:22.257545 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07244365-3feb-4c2c-a1bf-9c4b62eeff52 service neutron] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:00:22.257820 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07244365-3feb-4c2c-a1bf-9c4b62eeff52 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:00:22.267075 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07244365-3feb-4c2c-a1bf-9c4b62eeff52 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:00:22.268496 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07244365-3feb-4c2c-a1bf-9c4b62eeff52 service neutron] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:00:22.278012 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 761/1533] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Aug 30 02:00:22 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 37 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 02:00:25.622666 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b238f1ec-2fdd-4a09-ac6d-d805983e9912 None None] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:00:25.623267 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b238f1ec-2fdd-4a09-ac6d-d805983e9912 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:00:25.623267 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b238f1ec-2fdd-4a09-ac6d-d805983e9912 None None] PATH_INFO: `/` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:00:25.625067 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 773/1534] 127.0.0.1 () {54 vars in 917 bytes} [Tue Aug 30 02:00:25 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 02:00:25.630116 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af6f0735-6f21-4032-8057-810f41169303 None None] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:00:25.630528 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af6f0735-6f21-4032-8057-810f41169303 None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:00:25.630662 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af6f0735-6f21-4032-8057-810f41169303 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:00:25.638060 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=74 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:00:25.650136 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.auth.core [None req-af6f0735-6f21-4032-8057-810f41169303 None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81578) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:00:25.651338 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-af6f0735-6f21-4032-8057-810f41169303 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:00:25.659600 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 762/1535] 127.0.0.1 () {58 vars in 1014 bytes} [Tue Aug 30 02:00:25 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 02:00:26.210211 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b238f1ec-2fdd-4a09-ac6d-d805983e9912 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:00:26.212194 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-b238f1ec-2fdd-4a09-ac6d-d805983e9912 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:00:26.269012 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b238f1ec-2fdd-4a09-ac6d-d805983e9912 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:00:26.272505 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66d9311f-2b9d-4702-a346-50294911ffc7 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ab936b3767a8417aa80c8c8f1b84b667', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4562f3951fbe465280073e60a11e44e2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:00:26.273117 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66d9311f-2b9d-4702-a346-50294911ffc7 service cinder] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:00:26.273375 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66d9311f-2b9d-4702-a346-50294911ffc7 service cinder] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:00:26.273683 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66d9311f-2b9d-4702-a346-50294911ffc7 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:00:26.282380 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66d9311f-2b9d-4702-a346-50294911ffc7 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:00:26.283816 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66d9311f-2b9d-4702-a346-50294911ffc7 service cinder] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:00:26.293149 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 774/1536] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Aug 30 02:00:26 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 83 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 02:00:28.241667 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3dfe37d-b61c-418d-9e46-f7810121ae90 None None] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:00:28.242238 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3dfe37d-b61c-418d-9e46-f7810121ae90 None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:00:28.242238 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3dfe37d-b61c-418d-9e46-f7810121ae90 None None] PATH_INFO: `/` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:00:28.243541 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 763/1537] 127.0.0.1 () {54 vars in 917 bytes} [Tue Aug 30 02:00:28 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 02:00:28.249302 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4362b766-806b-4d15-9f05-fd16687eb6b9 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:00:28.249748 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4362b766-806b-4d15-9f05-fd16687eb6b9 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:00:28.249884 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4362b766-806b-4d15-9f05-fd16687eb6b9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:00:28.269049 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-4362b766-806b-4d15-9f05-fd16687eb6b9 None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:00:28.270269 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-4362b766-806b-4d15-9f05-fd16687eb6b9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:00:28.277898 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 775/1538] 127.0.0.1 () {58 vars in 1014 bytes} [Tue Aug 30 02:00:28 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 02:00:28.832725 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3dfe37d-b61c-418d-9e46-f7810121ae90 None None] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:00:28.834412 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-f3dfe37d-b61c-418d-9e46-f7810121ae90 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:00:28.848724 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3dfe37d-b61c-418d-9e46-f7810121ae90 None None] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:00:28.852251 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-beecd6b5-2409-44fe-82cc-6514e912fc07 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ab936b3767a8417aa80c8c8f1b84b667', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4562f3951fbe465280073e60a11e44e2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:00:28.852846 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-beecd6b5-2409-44fe-82cc-6514e912fc07 service cinder] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:00:28.853128 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-beecd6b5-2409-44fe-82cc-6514e912fc07 service cinder] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:00:28.853401 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-beecd6b5-2409-44fe-82cc-6514e912fc07 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:00:28.862561 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-beecd6b5-2409-44fe-82cc-6514e912fc07 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:00:28.863970 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-beecd6b5-2409-44fe-82cc-6514e912fc07 service cinder] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:00:28.873152 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 764/1539] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Aug 30 02:00:28 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 41 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 02:00:38.263765 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=62 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:00:38.869116 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=8 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:01:52.742523 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b441609-cc32-40ee-aca8-ecf8cc2e7d08 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:01:52.747002 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b441609-cc32-40ee-aca8-ecf8cc2e7d08 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:01:52.747002 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b441609-cc32-40ee-aca8-ecf8cc2e7d08 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:01:52.752685 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17541f08-1e1a-40aa-9862-e89ae499490c None None] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:01:52.753645 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17541f08-1e1a-40aa-9862-e89ae499490c None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:01:52.754135 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17541f08-1e1a-40aa-9862-e89ae499490c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:01:52.784353 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.auth.core [None req-17541f08-1e1a-40aa-9862-e89ae499490c None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81578) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:01:52.785928 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-17541f08-1e1a-40aa-9862-e89ae499490c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:01:52.789857 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-1b441609-cc32-40ee-aca8-ecf8cc2e7d08 None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:01:52.791158 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-1b441609-cc32-40ee-aca8-ecf8cc2e7d08 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:01:52.801870 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 765/1540] 10.222.0.27 () {58 vars in 958 bytes} [Tue Aug 30 02:01:52 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 50 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 30 02:01:52.804709 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 776/1541] 10.222.0.27 () {58 vars in 958 bytes} [Tue Aug 30 02:01:52 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 63 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 02:01:52.810825 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3255378-7757-41cb-acc6-b4c623a35f30 None None] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:01:52.811978 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3255378-7757-41cb-acc6-b4c623a35f30 None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:01:52.812641 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3255378-7757-41cb-acc6-b4c623a35f30 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:01:52.818657 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e00ebbf-7a38-4473-bec3-ced8710e196f None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:01:52.819059 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e00ebbf-7a38-4473-bec3-ced8710e196f None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:01:52.819442 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e00ebbf-7a38-4473-bec3-ced8710e196f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:01:52.846803 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.auth.core [None req-f3255378-7757-41cb-acc6-b4c623a35f30 None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81578) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:01:52.848337 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-f3255378-7757-41cb-acc6-b4c623a35f30 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:01:52.859067 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-6e00ebbf-7a38-4473-bec3-ced8710e196f None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:01:52.862368 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 766/1542] 10.222.0.27 () {58 vars in 958 bytes} [Tue Aug 30 02:01:52 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 52 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 02:01:52.863646 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-6e00ebbf-7a38-4473-bec3-ced8710e196f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:01:52.865173 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7b0f54f-7853-4f52-907d-9b35393dbf25 None None] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:01:52.866726 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7b0f54f-7853-4f52-907d-9b35393dbf25 None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:01:52.868149 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7b0f54f-7853-4f52-907d-9b35393dbf25 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:01:52.887564 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 777/1543] 10.222.0.27 () {58 vars in 958 bytes} [Tue Aug 30 02:01:52 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 74 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 30 02:01:52.891261 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3a4595d-5e59-41d5-933b-5649058ebb3e None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:01:52.893952 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3a4595d-5e59-41d5-933b-5649058ebb3e None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:01:52.894418 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3a4595d-5e59-41d5-933b-5649058ebb3e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:01:52.914804 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.auth.core [None req-b7b0f54f-7853-4f52-907d-9b35393dbf25 None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81578) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:01:52.917218 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-b7b0f54f-7853-4f52-907d-9b35393dbf25 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:01:52.937054 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 767/1544] 10.222.0.27 () {58 vars in 958 bytes} [Tue Aug 30 02:01:52 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 72 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 02:01:52.939459 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7b0f54f-7853-4f52-907d-9b35393dbf25 None None] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:01:52.940855 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-b3a4595d-5e59-41d5-933b-5649058ebb3e None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:01:52.942202 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-b3a4595d-5e59-41d5-933b-5649058ebb3e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:01:52.955682 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7b0f54f-7853-4f52-907d-9b35393dbf25 None None] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:01:52.963935 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-001cf4f8-41b6-42b8-beb2-27aa0ef9d2c1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:01:52.964572 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-001cf4f8-41b6-42b8-beb2-27aa0ef9d2c1 admin admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:01:52.965037 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-001cf4f8-41b6-42b8-beb2-27aa0ef9d2c1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:01:52.965309 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-001cf4f8-41b6-42b8-beb2-27aa0ef9d2c1 admin admin] PATH_INFO: `/v3/domains` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:01:52.966002 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-001cf4f8-41b6-42b8-beb2-27aa0ef9d2c1 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81578) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 02:01:52.966301 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-001cf4f8-41b6-42b8-beb2-27aa0ef9d2c1 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:01:52.966640 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 778/1545] 10.222.0.27 () {58 vars in 958 bytes} [Tue Aug 30 02:01:52 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 78 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 02:01:52.967915 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: WARNING py.warnings [None req-001cf4f8-41b6-42b8-beb2-27aa0ef9d2c1 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:list_domains": "role:reader and system_scope:all" failed scope check. The token used to make the request was project scoped but the policy requires ['system'] scope. This behavior may change in the future where using the intended scope is required Aug 30 02:01:52.967915 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: warnings.warn(msg) Aug 30 02:01:52.967915 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: Aug 30 02:01:52.968973 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3a4595d-5e59-41d5-933b-5649058ebb3e None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:01:52.969099 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-001cf4f8-41b6-42b8-beb2-27aa0ef9d2c1 admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:01:52.994738 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 768/1546] 10.222.0.27 () {58 vars in 1153 bytes} [Tue Aug 30 02:01:52 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 54 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 02:01:52.994939 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3a4595d-5e59-41d5-933b-5649058ebb3e None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:01:52.995061 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f910af41-d474-4463-bdb5-0c84214e7306 None None] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:01:52.995061 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f910af41-d474-4463-bdb5-0c84214e7306 None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:01:52.995061 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f910af41-d474-4463-bdb5-0c84214e7306 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:01:53.022053 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2263c074-1dd3-4540-9f22-20014fa78ac3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:01:53.022936 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2263c074-1dd3-4540-9f22-20014fa78ac3 admin admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:01:53.023219 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2263c074-1dd3-4540-9f22-20014fa78ac3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:01:53.023495 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2263c074-1dd3-4540-9f22-20014fa78ac3 admin admin] PATH_INFO: `/v3/domains` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:01:53.024159 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2263c074-1dd3-4540-9f22-20014fa78ac3 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 02:01:53.024642 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2263c074-1dd3-4540-9f22-20014fa78ac3 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:01:53.026099 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.auth.core [None req-f910af41-d474-4463-bdb5-0c84214e7306 None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81578) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:01:53.026828 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2263c074-1dd3-4540-9f22-20014fa78ac3 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:01:53.027326 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-f910af41-d474-4463-bdb5-0c84214e7306 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:01:53.042202 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 769/1547] 10.222.0.27 () {58 vars in 958 bytes} [Tue Aug 30 02:01:52 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 50 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 02:01:53.044407 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9b4c5be-c038-4f16-9357-fe0bfb5ec297 None None] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:01:53.044716 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9b4c5be-c038-4f16-9357-fe0bfb5ec297 None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:01:53.044970 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9b4c5be-c038-4f16-9357-fe0bfb5ec297 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:01:53.054442 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 779/1548] 10.222.0.27 () {58 vars in 1153 bytes} [Tue Aug 30 02:01:52 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 87 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 02:01:53.058092 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a681531-ad02-498b-8625-ae1fc1d784a5 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:01:53.058092 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a681531-ad02-498b-8625-ae1fc1d784a5 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:01:53.058092 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a681531-ad02-498b-8625-ae1fc1d784a5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:01:53.074459 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.auth.core [None req-b9b4c5be-c038-4f16-9357-fe0bfb5ec297 None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81578) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:01:53.075687 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-b9b4c5be-c038-4f16-9357-fe0bfb5ec297 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:01:53.088932 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 770/1549] 10.222.0.27 () {58 vars in 958 bytes} [Tue Aug 30 02:01:53 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 46 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 02:01:53.095808 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9b4c5be-c038-4f16-9357-fe0bfb5ec297 None None] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:01:53.095950 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-6a681531-ad02-498b-8625-ae1fc1d784a5 None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:01:53.095950 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-6a681531-ad02-498b-8625-ae1fc1d784a5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:01:53.101354 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9b4c5be-c038-4f16-9357-fe0bfb5ec297 None None] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:01:53.101693 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 780/1550] 10.222.0.27 () {58 vars in 958 bytes} [Tue Aug 30 02:01:53 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 46 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 02:01:53.102463 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a681531-ad02-498b-8625-ae1fc1d784a5 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:01:53.109611 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba51ead4-0a64-4731-941d-7cbb792064ba admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:01:53.110575 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba51ead4-0a64-4731-941d-7cbb792064ba admin admin] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:01:53.110853 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba51ead4-0a64-4731-941d-7cbb792064ba admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:01:53.111170 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba51ead4-0a64-4731-941d-7cbb792064ba admin admin] PATH_INFO: `/v3/projects` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:01:53.111807 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba51ead4-0a64-4731-941d-7cbb792064ba admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:01:53.113811 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba51ead4-0a64-4731-941d-7cbb792064ba admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:01:53.115258 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a681531-ad02-498b-8625-ae1fc1d784a5 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:01:53.119951 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-638d5ac1-1d23-43f8-a38d-59eeed4ada03 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:01:53.120546 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-638d5ac1-1d23-43f8-a38d-59eeed4ada03 admin admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:01:53.120816 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-638d5ac1-1d23-43f8-a38d-59eeed4ada03 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:01:53.121089 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-638d5ac1-1d23-43f8-a38d-59eeed4ada03 admin admin] PATH_INFO: `/v3/domains` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:01:53.121764 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-638d5ac1-1d23-43f8-a38d-59eeed4ada03 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 02:01:53.122067 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-638d5ac1-1d23-43f8-a38d-59eeed4ada03 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:01:53.124231 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-638d5ac1-1d23-43f8-a38d-59eeed4ada03 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:01:53.135875 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 781/1551] 10.222.0.27 () {58 vars in 1153 bytes} [Tue Aug 30 02:01:53 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 02:01:53.137075 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05c2f328-c8f0-4327-aeaf-5cf439c3e5c2 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:01:53.137368 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05c2f328-c8f0-4327-aeaf-5cf439c3e5c2 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:01:53.137667 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05c2f328-c8f0-4327-aeaf-5cf439c3e5c2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:01:53.156751 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-05c2f328-c8f0-4327-aeaf-5cf439c3e5c2 None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:01:53.158763 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-05c2f328-c8f0-4327-aeaf-5cf439c3e5c2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:01:53.165843 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 782/1552] 10.222.0.27 () {58 vars in 958 bytes} [Tue Aug 30 02:01:53 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 29 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 02:01:53.166995 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9690d534-713f-4f59-84f5-faf363671305 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:01:53.167274 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9690d534-713f-4f59-84f5-faf363671305 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:01:53.167542 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9690d534-713f-4f59-84f5-faf363671305 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:01:53.187750 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-9690d534-713f-4f59-84f5-faf363671305 None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:01:53.188937 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-9690d534-713f-4f59-84f5-faf363671305 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:01:53.196691 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 783/1553] 10.222.0.27 () {58 vars in 958 bytes} [Tue Aug 30 02:01:53 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 02:01:53.197502 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9690d534-713f-4f59-84f5-faf363671305 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:01:53.206992 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9690d534-713f-4f59-84f5-faf363671305 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:01:53.211660 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b25a005e-3f75-40ba-8033-93c790a0cf92 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:01:53.212261 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b25a005e-3f75-40ba-8033-93c790a0cf92 admin admin] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:01:53.212542 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b25a005e-3f75-40ba-8033-93c790a0cf92 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:01:53.212815 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b25a005e-3f75-40ba-8033-93c790a0cf92 admin admin] PATH_INFO: `/v3/projects` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:01:53.213444 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b25a005e-3f75-40ba-8033-93c790a0cf92 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:01:53.214494 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: WARNING py.warnings [None req-b25a005e-3f75-40ba-8033-93c790a0cf92 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_project": "(role:admin and system_scope:all) or (role:admin and domain_id:%(target.project.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Aug 30 02:01:53.214494 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: warnings.warn(msg) Aug 30 02:01:53.214494 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: Aug 30 02:01:53.215683 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b25a005e-3f75-40ba-8033-93c790a0cf92 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:01:53.248690 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 771/1554] 10.222.0.27 () {60 vars in 1153 bytes} [Tue Aug 30 02:01:53 2022] POST /identity/v3/projects => generated 379 bytes in 159 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 02:01:53.250602 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1a8add9-a36f-4ff1-887d-7e755d2f5745 None None] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:01:53.251005 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1a8add9-a36f-4ff1-887d-7e755d2f5745 None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:01:53.251280 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1a8add9-a36f-4ff1-887d-7e755d2f5745 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:01:53.271203 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.auth.core [None req-f1a8add9-a36f-4ff1-887d-7e755d2f5745 None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81578) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:01:53.272426 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-f1a8add9-a36f-4ff1-887d-7e755d2f5745 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:01:53.321105 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 784/1555] 10.222.0.27 () {60 vars in 1153 bytes} [Tue Aug 30 02:01:53 2022] POST /identity/v3/projects => generated 373 bytes in 121 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 02:01:53.321105 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b25a005e-3f75-40ba-8033-93c790a0cf92 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:01:53.331648 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 772/1556] 10.222.0.27 () {58 vars in 958 bytes} [Tue Aug 30 02:01:53 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 02:01:53.334754 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccb78b94-54f0-464d-ac0f-c23bf02db563 None None] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:01:53.335258 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccb78b94-54f0-464d-ac0f-c23bf02db563 None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:01:53.335539 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccb78b94-54f0-464d-ac0f-c23bf02db563 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:01:53.356413 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.auth.core [None req-ccb78b94-54f0-464d-ac0f-c23bf02db563 None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81578) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:01:53.357662 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-ccb78b94-54f0-464d-ac0f-c23bf02db563 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:01:53.382497 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b25a005e-3f75-40ba-8033-93c790a0cf92 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:01:53.387021 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6dd54c0d-cbed-414c-b309-22048c800a67 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:01:53.387622 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dd54c0d-cbed-414c-b309-22048c800a67 admin admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:01:53.387898 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dd54c0d-cbed-414c-b309-22048c800a67 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:01:53.388521 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dd54c0d-cbed-414c-b309-22048c800a67 admin admin] PATH_INFO: `/v3/domains` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:01:53.389162 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6dd54c0d-cbed-414c-b309-22048c800a67 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 02:01:53.389480 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6dd54c0d-cbed-414c-b309-22048c800a67 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:01:53.391081 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6dd54c0d-cbed-414c-b309-22048c800a67 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:01:53.402562 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 785/1557] 10.222.0.27 () {58 vars in 1153 bytes} [Tue Aug 30 02:01:53 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 84 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 02:01:53.402562 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6dd54c0d-cbed-414c-b309-22048c800a67 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:01:53.412987 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6dd54c0d-cbed-414c-b309-22048c800a67 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:01:53.417827 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c59bab0b-bae9-4f0e-af0b-2129523bcfe7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:01:53.418409 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c59bab0b-bae9-4f0e-af0b-2129523bcfe7 admin admin] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:01:53.418682 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c59bab0b-bae9-4f0e-af0b-2129523bcfe7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:01:53.418949 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c59bab0b-bae9-4f0e-af0b-2129523bcfe7 admin admin] PATH_INFO: `/v3/projects` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:01:53.419559 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c59bab0b-bae9-4f0e-af0b-2129523bcfe7 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:01:53.422140 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c59bab0b-bae9-4f0e-af0b-2129523bcfe7 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:01:53.424303 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 773/1558] 10.222.0.27 () {58 vars in 958 bytes} [Tue Aug 30 02:01:53 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 02:01:53.425579 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-504b9cb4-eeaf-4abc-b6a2-e9d00a2d2907 None None] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:01:53.425869 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-504b9cb4-eeaf-4abc-b6a2-e9d00a2d2907 None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:01:53.426142 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-504b9cb4-eeaf-4abc-b6a2-e9d00a2d2907 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:01:53.446045 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.auth.core [None req-504b9cb4-eeaf-4abc-b6a2-e9d00a2d2907 None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81578) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:01:53.447266 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-504b9cb4-eeaf-4abc-b6a2-e9d00a2d2907 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:01:53.455760 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 774/1559] 10.222.0.27 () {58 vars in 958 bytes} [Tue Aug 30 02:01:53 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 02:01:53.456994 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb772f8a-de45-4998-8913-f6be285ac8d8 None None] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:01:53.457298 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb772f8a-de45-4998-8913-f6be285ac8d8 None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:01:53.457596 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb772f8a-de45-4998-8913-f6be285ac8d8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:01:53.477023 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.auth.core [None req-cb772f8a-de45-4998-8913-f6be285ac8d8 None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81578) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:01:53.478201 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-cb772f8a-de45-4998-8913-f6be285ac8d8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:01:53.486400 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 775/1560] 10.222.0.27 () {58 vars in 958 bytes} [Tue Aug 30 02:01:53 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 02:01:53.487121 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb772f8a-de45-4998-8913-f6be285ac8d8 None None] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:01:53.496242 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb772f8a-de45-4998-8913-f6be285ac8d8 None None] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:01:53.501026 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b97d5b0-5475-40ab-9e06-6abe7a2fe185 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:01:53.501557 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b97d5b0-5475-40ab-9e06-6abe7a2fe185 admin admin] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:01:53.501834 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b97d5b0-5475-40ab-9e06-6abe7a2fe185 admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:01:53.502102 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b97d5b0-5475-40ab-9e06-6abe7a2fe185 admin admin] PATH_INFO: `/v3/users` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:01:53.502762 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b97d5b0-5475-40ab-9e06-6abe7a2fe185 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:01:53.503739 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: WARNING py.warnings [None req-1b97d5b0-5475-40ab-9e06-6abe7a2fe185 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_user": "(role:admin and system_scope:all) or (role:admin and token.domain.id:%(target.user.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Aug 30 02:01:53.503739 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: warnings.warn(msg) Aug 30 02:01:53.503739 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: Aug 30 02:01:53.504913 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b97d5b0-5475-40ab-9e06-6abe7a2fe185 admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:01:53.511106 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 786/1561] 10.222.0.27 () {60 vars in 1153 bytes} [Tue Aug 30 02:01:53 2022] POST /identity/v3/projects => generated 377 bytes in 108 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 02:01:53.512029 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c59bab0b-bae9-4f0e-af0b-2129523bcfe7 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:01:53.515223 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:01:53.559298 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:01:53.574916 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c59bab0b-bae9-4f0e-af0b-2129523bcfe7 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:01:53.579874 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc38e4ce-b7f7-4592-b454-e149dcd765db admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:01:53.582557 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc38e4ce-b7f7-4592-b454-e149dcd765db admin admin] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:01:53.582557 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc38e4ce-b7f7-4592-b454-e149dcd765db admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:01:53.582557 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc38e4ce-b7f7-4592-b454-e149dcd765db admin admin] PATH_INFO: `/v3/users` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:01:53.582557 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc38e4ce-b7f7-4592-b454-e149dcd765db admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:01:53.583791 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc38e4ce-b7f7-4592-b454-e149dcd765db admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:01:53.630109 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 776/1562] 10.222.0.27 () {60 vars in 1144 bytes} [Tue Aug 30 02:01:53 2022] POST /identity/v3/users => generated 290 bytes in 143 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 02:01:53.631384 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfcae46c-07fd-4dfd-9090-38be457c307c None None] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:01:53.631677 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfcae46c-07fd-4dfd-9090-38be457c307c None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:01:53.631949 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfcae46c-07fd-4dfd-9090-38be457c307c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:01:53.650538 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.auth.core [None req-dfcae46c-07fd-4dfd-9090-38be457c307c None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81578) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:01:53.651768 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-dfcae46c-07fd-4dfd-9090-38be457c307c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:01:53.660815 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 777/1563] 10.222.0.27 () {58 vars in 958 bytes} [Tue Aug 30 02:01:53 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 02:01:53.661718 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfcae46c-07fd-4dfd-9090-38be457c307c None None] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:01:53.671421 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfcae46c-07fd-4dfd-9090-38be457c307c None None] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:01:53.675954 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52f1d819-9553-4b3b-96ef-9b1a815328b1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:01:53.685969 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52f1d819-9553-4b3b-96ef-9b1a815328b1 admin admin] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:01:53.686152 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 787/1564] 10.222.0.27 () {60 vars in 1144 bytes} [Tue Aug 30 02:01:53 2022] POST /identity/v3/users => generated 287 bytes in 165 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 02:01:53.686367 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52f1d819-9553-4b3b-96ef-9b1a815328b1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:01:53.686367 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52f1d819-9553-4b3b-96ef-9b1a815328b1 admin admin] PATH_INFO: `/v3/projects` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:01:53.686566 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc38e4ce-b7f7-4592-b454-e149dcd765db admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:01:53.686658 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52f1d819-9553-4b3b-96ef-9b1a815328b1 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:01:53.686658 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52f1d819-9553-4b3b-96ef-9b1a815328b1 admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:01:53.690388 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc38e4ce-b7f7-4592-b454-e149dcd765db admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:01:53.695043 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a6ec1e6-50c9-4fab-a7ed-60a69eb31b1c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:01:53.695638 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a6ec1e6-50c9-4fab-a7ed-60a69eb31b1c admin admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:01:53.695916 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a6ec1e6-50c9-4fab-a7ed-60a69eb31b1c admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:01:53.696182 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a6ec1e6-50c9-4fab-a7ed-60a69eb31b1c admin admin] PATH_INFO: `/v3/domains` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:01:53.696817 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a6ec1e6-50c9-4fab-a7ed-60a69eb31b1c admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 02:01:53.697265 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a6ec1e6-50c9-4fab-a7ed-60a69eb31b1c admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:01:53.699723 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a6ec1e6-50c9-4fab-a7ed-60a69eb31b1c admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:01:53.712436 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 788/1565] 10.222.0.27 () {58 vars in 1153 bytes} [Tue Aug 30 02:01:53 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 02:01:53.713262 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a6ec1e6-50c9-4fab-a7ed-60a69eb31b1c admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:01:53.725089 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a6ec1e6-50c9-4fab-a7ed-60a69eb31b1c admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:01:53.729680 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91b6a546-452c-4d75-a68b-fbcc839e7c49 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:01:53.730277 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91b6a546-452c-4d75-a68b-fbcc839e7c49 admin admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:01:53.730553 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91b6a546-452c-4d75-a68b-fbcc839e7c49 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:01:53.730821 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91b6a546-452c-4d75-a68b-fbcc839e7c49 admin admin] PATH_INFO: `/v3/domains` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:01:53.731460 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91b6a546-452c-4d75-a68b-fbcc839e7c49 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 02:01:53.731760 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91b6a546-452c-4d75-a68b-fbcc839e7c49 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:01:53.733938 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91b6a546-452c-4d75-a68b-fbcc839e7c49 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:01:53.745204 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 789/1566] 10.222.0.27 () {58 vars in 1153 bytes} [Tue Aug 30 02:01:53 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 02:01:53.746023 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91b6a546-452c-4d75-a68b-fbcc839e7c49 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:01:53.759314 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91b6a546-452c-4d75-a68b-fbcc839e7c49 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:01:53.760042 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e3af30c-fe6d-4ced-b3fd-231b45f328dd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:01:53.760614 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e3af30c-fe6d-4ced-b3fd-231b45f328dd admin admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:01:53.760893 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e3af30c-fe6d-4ced-b3fd-231b45f328dd admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:01:53.761165 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e3af30c-fe6d-4ced-b3fd-231b45f328dd admin admin] PATH_INFO: `/v3/domains` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:01:53.761829 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e3af30c-fe6d-4ced-b3fd-231b45f328dd admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 02:01:53.762132 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e3af30c-fe6d-4ced-b3fd-231b45f328dd admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:01:53.763702 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e3af30c-fe6d-4ced-b3fd-231b45f328dd admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:01:53.775256 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 790/1567] 10.222.0.27 () {58 vars in 1153 bytes} [Tue Aug 30 02:01:53 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 02:01:53.776066 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e3af30c-fe6d-4ced-b3fd-231b45f328dd admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:01:53.786020 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e3af30c-fe6d-4ced-b3fd-231b45f328dd admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:01:53.786888 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 778/1568] 10.222.0.27 () {60 vars in 1153 bytes} [Tue Aug 30 02:01:53 2022] POST /identity/v3/projects => generated 377 bytes in 126 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 02:01:53.788012 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52f1d819-9553-4b3b-96ef-9b1a815328b1 admin admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:01:53.845286 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35c0fb3c-04cf-4244-be0c-3c59767b4ec6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:01:53.845926 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35c0fb3c-04cf-4244-be0c-3c59767b4ec6 admin admin] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:01:53.846206 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35c0fb3c-04cf-4244-be0c-3c59767b4ec6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:01:53.846475 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35c0fb3c-04cf-4244-be0c-3c59767b4ec6 admin admin] PATH_INFO: `/v3/users` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:01:53.847098 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35c0fb3c-04cf-4244-be0c-3c59767b4ec6 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:01:53.849002 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35c0fb3c-04cf-4244-be0c-3c59767b4ec6 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:01:53.851573 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52f1d819-9553-4b3b-96ef-9b1a815328b1 admin admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:01:53.856165 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5176cfad-eb04-4caf-9274-59ee872f5b02 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:01:53.856769 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5176cfad-eb04-4caf-9274-59ee872f5b02 admin admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:01:53.857053 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5176cfad-eb04-4caf-9274-59ee872f5b02 admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:01:53.857321 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5176cfad-eb04-4caf-9274-59ee872f5b02 admin admin] PATH_INFO: `/v3/roles` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:01:53.858062 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5176cfad-eb04-4caf-9274-59ee872f5b02 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:01:53.859635 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5176cfad-eb04-4caf-9274-59ee872f5b02 admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:01:53.869944 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 779/1569] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Aug 30 02:01:53 2022] GET /identity/v3/roles => generated 1420 bytes in 82 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 02:01:53.873367 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5176cfad-eb04-4caf-9274-59ee872f5b02 admin admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:01:53.880207 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5176cfad-eb04-4caf-9274-59ee872f5b02 admin admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:01:53.884772 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90fdc09f-81e5-40c3-9f1c-673d7f9bae31 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:01:53.885363 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90fdc09f-81e5-40c3-9f1c-673d7f9bae31 admin admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:01:53.885675 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90fdc09f-81e5-40c3-9f1c-673d7f9bae31 admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:01:53.885949 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90fdc09f-81e5-40c3-9f1c-673d7f9bae31 admin admin] PATH_INFO: `/v3/domains` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:01:53.886570 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90fdc09f-81e5-40c3-9f1c-673d7f9bae31 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81578) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 02:01:53.886868 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90fdc09f-81e5-40c3-9f1c-673d7f9bae31 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:01:53.888422 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90fdc09f-81e5-40c3-9f1c-673d7f9bae31 admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:01:53.900691 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 780/1570] 10.222.0.27 () {58 vars in 1153 bytes} [Tue Aug 30 02:01:53 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 02:01:53.901972 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90fdc09f-81e5-40c3-9f1c-673d7f9bae31 admin admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:01:53.911583 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90fdc09f-81e5-40c3-9f1c-673d7f9bae31 admin admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:01:53.916174 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a58831e-83cb-4c07-80e5-29be50697935 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:01:53.916737 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a58831e-83cb-4c07-80e5-29be50697935 admin admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:01:53.917016 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a58831e-83cb-4c07-80e5-29be50697935 admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:01:53.917290 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a58831e-83cb-4c07-80e5-29be50697935 admin admin] PATH_INFO: `/v3/roles` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:01:53.917936 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a58831e-83cb-4c07-80e5-29be50697935 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:01:53.919468 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a58831e-83cb-4c07-80e5-29be50697935 admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:01:53.929057 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 781/1571] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Aug 30 02:01:53 2022] GET /identity/v3/roles => generated 1420 bytes in 28 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 02:01:53.929829 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a58831e-83cb-4c07-80e5-29be50697935 admin admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:01:53.938978 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a58831e-83cb-4c07-80e5-29be50697935 admin admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:01:53.943488 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-784fb990-1de2-4b43-943d-dac164b8a4db admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:01:53.944036 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-784fb990-1de2-4b43-943d-dac164b8a4db admin admin] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:01:53.944316 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-784fb990-1de2-4b43-943d-dac164b8a4db admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:01:53.944585 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-784fb990-1de2-4b43-943d-dac164b8a4db admin admin] PATH_INFO: `/v3/projects` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:01:53.945185 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-784fb990-1de2-4b43-943d-dac164b8a4db admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:01:53.947112 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-784fb990-1de2-4b43-943d-dac164b8a4db admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:01:53.956547 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 791/1572] 10.222.0.27 () {60 vars in 1144 bytes} [Tue Aug 30 02:01:53 2022] POST /identity/v3/users => generated 289 bytes in 181 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 02:01:53.957525 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35c0fb3c-04cf-4244-be0c-3c59767b4ec6 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:01:53.967820 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35c0fb3c-04cf-4244-be0c-3c59767b4ec6 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:01:53.972447 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6727158-5057-43a2-86cd-51ec0b3bcdb1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:01:53.973033 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6727158-5057-43a2-86cd-51ec0b3bcdb1 admin admin] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:01:53.973308 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6727158-5057-43a2-86cd-51ec0b3bcdb1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:01:53.973637 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6727158-5057-43a2-86cd-51ec0b3bcdb1 admin admin] PATH_INFO: `/v3/projects` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:01:53.974254 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6727158-5057-43a2-86cd-51ec0b3bcdb1 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:01:53.976131 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6727158-5057-43a2-86cd-51ec0b3bcdb1 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:01:54.065451 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 782/1573] 10.222.0.27 () {60 vars in 1153 bytes} [Tue Aug 30 02:01:53 2022] POST /identity/v3/projects => generated 391 bytes in 136 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 02:01:54.066425 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-784fb990-1de2-4b43-943d-dac164b8a4db admin admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:01:54.080860 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 792/1574] 10.222.0.27 () {60 vars in 1153 bytes} [Tue Aug 30 02:01:53 2022] POST /identity/v3/projects => generated 385 bytes in 122 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 02:01:54.080860 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6727158-5057-43a2-86cd-51ec0b3bcdb1 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:01:54.132943 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-784fb990-1de2-4b43-943d-dac164b8a4db admin admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:01:54.142426 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6727158-5057-43a2-86cd-51ec0b3bcdb1 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:01:54.147219 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4629bf1c-29ce-4514-860a-fa2f19fe4e7b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:01:54.147900 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4629bf1c-29ce-4514-860a-fa2f19fe4e7b admin admin] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:01:54.148252 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4629bf1c-29ce-4514-860a-fa2f19fe4e7b admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:01:54.148595 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4629bf1c-29ce-4514-860a-fa2f19fe4e7b admin admin] PATH_INFO: `/v3/users` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:01:54.149290 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4629bf1c-29ce-4514-860a-fa2f19fe4e7b admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:01:54.151245 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:INSERT=3,keystone:SELECT=97 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:01:54.151370 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4629bf1c-29ce-4514-860a-fa2f19fe4e7b admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:01:54.160877 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=92,keystone:INSERT=8 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:01:54.190423 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0641e99-8edc-4297-966c-498263b41903 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:01:54.191022 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0641e99-8edc-4297-966c-498263b41903 admin admin] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:01:54.191298 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0641e99-8edc-4297-966c-498263b41903 admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:01:54.191563 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0641e99-8edc-4297-966c-498263b41903 admin admin] PATH_INFO: `/v3/projects` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:01:54.192178 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0641e99-8edc-4297-966c-498263b41903 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:01:54.194123 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0641e99-8edc-4297-966c-498263b41903 admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:01:54.265607 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 793/1575] 10.222.0.27 () {60 vars in 1144 bytes} [Tue Aug 30 02:01:54 2022] POST /identity/v3/users => generated 289 bytes in 186 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 02:01:54.266544 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4629bf1c-29ce-4514-860a-fa2f19fe4e7b admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:01:54.275699 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4629bf1c-29ce-4514-860a-fa2f19fe4e7b admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:01:54.280318 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d851a7c2-80aa-437e-a527-cb7b41c3916f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:01:54.280888 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d851a7c2-80aa-437e-a527-cb7b41c3916f admin admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:01:54.281181 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d851a7c2-80aa-437e-a527-cb7b41c3916f admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:01:54.281479 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d851a7c2-80aa-437e-a527-cb7b41c3916f admin admin] PATH_INFO: `/v3/roles` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:01:54.282112 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d851a7c2-80aa-437e-a527-cb7b41c3916f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:01:54.283700 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d851a7c2-80aa-437e-a527-cb7b41c3916f admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:01:54.293509 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 794/1576] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Aug 30 02:01:54 2022] GET /identity/v3/roles => generated 1420 bytes in 27 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 02:01:54.296218 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d851a7c2-80aa-437e-a527-cb7b41c3916f admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:01:54.300103 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 783/1577] 10.222.0.27 () {60 vars in 1153 bytes} [Tue Aug 30 02:01:54 2022] POST /identity/v3/projects => generated 371 bytes in 231 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 02:01:54.300103 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0641e99-8edc-4297-966c-498263b41903 admin admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:01:54.303918 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d851a7c2-80aa-437e-a527-cb7b41c3916f admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:01:54.359348 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0641e99-8edc-4297-966c-498263b41903 admin admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:01:54.361288 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e406a326-4475-4ce2-9ad1-5d2c106b1f3e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:01:54.361932 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e406a326-4475-4ce2-9ad1-5d2c106b1f3e admin admin] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:01:54.362214 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e406a326-4475-4ce2-9ad1-5d2c106b1f3e admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:01:54.364829 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e406a326-4475-4ce2-9ad1-5d2c106b1f3e admin admin] PATH_INFO: `/v3/projects` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:01:54.364829 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e406a326-4475-4ce2-9ad1-5d2c106b1f3e admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:01:54.365042 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-219507e8-ee3e-42f0-9708-a95c6d78f4b8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:01:54.365042 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-219507e8-ee3e-42f0-9708-a95c6d78f4b8 admin admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:01:54.365249 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e406a326-4475-4ce2-9ad1-5d2c106b1f3e admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:01:54.365347 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-219507e8-ee3e-42f0-9708-a95c6d78f4b8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:01:54.365703 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-219507e8-ee3e-42f0-9708-a95c6d78f4b8 admin admin] PATH_INFO: `/v3/roles` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:01:54.366372 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-219507e8-ee3e-42f0-9708-a95c6d78f4b8 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:01:54.367961 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-219507e8-ee3e-42f0-9708-a95c6d78f4b8 admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:01:54.382987 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 784/1578] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Aug 30 02:01:54 2022] GET /identity/v3/roles => generated 1420 bytes in 83 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 02:01:54.382987 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-219507e8-ee3e-42f0-9708-a95c6d78f4b8 admin admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:01:54.394490 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-219507e8-ee3e-42f0-9708-a95c6d78f4b8 admin admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:01:54.399483 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-452ae72d-6fae-4538-a4df-7d4ca2de8cca admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:01:54.400267 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-452ae72d-6fae-4538-a4df-7d4ca2de8cca admin admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:01:54.400545 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-452ae72d-6fae-4538-a4df-7d4ca2de8cca admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:01:54.400995 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-452ae72d-6fae-4538-a4df-7d4ca2de8cca admin admin] PATH_INFO: `/v3/roles` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:01:54.401516 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-452ae72d-6fae-4538-a4df-7d4ca2de8cca admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:01:54.403414 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-452ae72d-6fae-4538-a4df-7d4ca2de8cca admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:01:54.419579 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 785/1579] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Aug 30 02:01:54 2022] GET /identity/v3/roles => generated 1420 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 02:01:54.420604 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-452ae72d-6fae-4538-a4df-7d4ca2de8cca admin admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:01:54.433390 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-452ae72d-6fae-4538-a4df-7d4ca2de8cca admin admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:01:54.438694 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-297370c1-0081-4725-92c4-356100d206d6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:01:54.439293 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-297370c1-0081-4725-92c4-356100d206d6 admin admin] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:01:54.439570 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-297370c1-0081-4725-92c4-356100d206d6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:01:54.439838 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-297370c1-0081-4725-92c4-356100d206d6 admin admin] PATH_INFO: `/v3/users` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:01:54.440491 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-297370c1-0081-4725-92c4-356100d206d6 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:01:54.442431 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-297370c1-0081-4725-92c4-356100d206d6 admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:01:54.479779 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 795/1580] 10.222.0.27 () {60 vars in 1153 bytes} [Tue Aug 30 02:01:54 2022] POST /identity/v3/projects => generated 385 bytes in 186 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 02:01:54.480604 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e406a326-4475-4ce2-9ad1-5d2c106b1f3e admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:01:54.533093 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 786/1581] 10.222.0.27 () {60 vars in 1144 bytes} [Tue Aug 30 02:01:54 2022] POST /identity/v3/users => generated 296 bytes in 113 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 02:01:54.533953 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-297370c1-0081-4725-92c4-356100d206d6 admin admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:01:54.543350 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-297370c1-0081-4725-92c4-356100d206d6 admin admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:01:54.546091 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e406a326-4475-4ce2-9ad1-5d2c106b1f3e admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:01:54.548264 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f9ba996-5542-416d-8dfb-51e1168fcb4e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:01:54.548822 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f9ba996-5542-416d-8dfb-51e1168fcb4e admin admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:01:54.549103 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f9ba996-5542-416d-8dfb-51e1168fcb4e admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:01:54.549373 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f9ba996-5542-416d-8dfb-51e1168fcb4e admin admin] PATH_INFO: `/v3/roles` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:01:54.550036 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f9ba996-5542-416d-8dfb-51e1168fcb4e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:01:54.551023 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a369bc1-bd54-44f2-a893-921c622f83c3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:01:54.551641 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f9ba996-5542-416d-8dfb-51e1168fcb4e admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:01:54.551765 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a369bc1-bd54-44f2-a893-921c622f83c3 admin admin] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:01:54.552020 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a369bc1-bd54-44f2-a893-921c622f83c3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:01:54.552293 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a369bc1-bd54-44f2-a893-921c622f83c3 admin admin] PATH_INFO: `/v3/users` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:01:54.552919 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a369bc1-bd54-44f2-a893-921c622f83c3 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:01:54.554866 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a369bc1-bd54-44f2-a893-921c622f83c3 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:01:54.563770 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 787/1582] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Aug 30 02:01:54 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 02:01:54.564807 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f9ba996-5542-416d-8dfb-51e1168fcb4e admin admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:01:54.575452 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f9ba996-5542-416d-8dfb-51e1168fcb4e admin admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:01:54.581026 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb6978a9-eae6-429e-b8f7-911d39560d79 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:01:54.581725 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb6978a9-eae6-429e-b8f7-911d39560d79 admin admin] REQUEST_METHOD: `PUT` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:01:54.584014 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb6978a9-eae6-429e-b8f7-911d39560d79 admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:01:54.584014 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb6978a9-eae6-429e-b8f7-911d39560d79 admin admin] PATH_INFO: `/v3/projects/1b77ab6a0dc743f993aa7cfe813a88ba/users/43fc688221d54c3badbb8a6ba3b6d9e3/roles/1fb81afcb9f245f3b989add94598a063` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:01:54.599156 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb6978a9-eae6-429e-b8f7-911d39560d79 admin admin] RBAC: Authorizing `identity:create_grant(project_id=1b77ab6a0dc743f993aa7cfe813a88ba, user_id=43fc688221d54c3badbb8a6ba3b6d9e3, role_id=1fb81afcb9f245f3b989add94598a063)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:01:54.601954 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb6978a9-eae6-429e-b8f7-911d39560d79 admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:01:54.636336 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 796/1583] 10.222.0.27 () {60 vars in 1144 bytes} [Tue Aug 30 02:01:54 2022] POST /identity/v3/users => generated 293 bytes in 156 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 02:01:54.637119 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a369bc1-bd54-44f2-a893-921c622f83c3 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:01:54.648714 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a369bc1-bd54-44f2-a893-921c622f83c3 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:01:54.655728 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-203f0b03-bb91-461a-bb8d-e5102c076588 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:01:54.656491 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-203f0b03-bb91-461a-bb8d-e5102c076588 admin admin] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:01:54.656765 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-203f0b03-bb91-461a-bb8d-e5102c076588 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:01:54.657068 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-203f0b03-bb91-461a-bb8d-e5102c076588 admin admin] PATH_INFO: `/v3/users` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:01:54.657739 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-203f0b03-bb91-461a-bb8d-e5102c076588 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:01:54.659725 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-203f0b03-bb91-461a-bb8d-e5102c076588 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:01:54.663316 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 788/1584] 10.222.0.27 () {60 vars in 1483 bytes} [Tue Aug 30 02:01:54 2022] PUT /identity/v3/projects/1b77ab6a0dc743f993aa7cfe813a88ba/users/43fc688221d54c3badbb8a6ba3b6d9e3/roles/1fb81afcb9f245f3b989add94598a063 => generated 0 bytes in 98 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 02:01:54.663316 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb6978a9-eae6-429e-b8f7-911d39560d79 admin admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:01:54.714103 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:01:54.737028 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb6978a9-eae6-429e-b8f7-911d39560d79 admin admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:01:54.741971 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3417cb3e-be28-4018-aa37-3aee07a1556a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:01:54.742774 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3417cb3e-be28-4018-aa37-3aee07a1556a admin admin] REQUEST_METHOD: `PUT` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:01:54.742995 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3417cb3e-be28-4018-aa37-3aee07a1556a admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:01:54.743269 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3417cb3e-be28-4018-aa37-3aee07a1556a admin admin] PATH_INFO: `/v3/projects/c28e611dd60f47dbb3a396a2b59406c7/users/8019b993d3b74f13ad20564a86f39ab3/roles/1fb81afcb9f245f3b989add94598a063` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:01:54.760863 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3417cb3e-be28-4018-aa37-3aee07a1556a admin admin] RBAC: Authorizing `identity:create_grant(project_id=c28e611dd60f47dbb3a396a2b59406c7, user_id=8019b993d3b74f13ad20564a86f39ab3, role_id=1fb81afcb9f245f3b989add94598a063)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:01:54.762342 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 797/1585] 10.222.0.27 () {60 vars in 1144 bytes} [Tue Aug 30 02:01:54 2022] POST /identity/v3/users => generated 286 bytes in 126 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 02:01:54.763246 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-203f0b03-bb91-461a-bb8d-e5102c076588 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:01:54.763395 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3417cb3e-be28-4018-aa37-3aee07a1556a admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:01:54.773502 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-203f0b03-bb91-461a-bb8d-e5102c076588 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:01:54.778240 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37787940-0b0a-41ed-8ea1-e9ca1e64c467 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:01:54.778847 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37787940-0b0a-41ed-8ea1-e9ca1e64c467 admin admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:01:54.779131 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37787940-0b0a-41ed-8ea1-e9ca1e64c467 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:01:54.779404 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37787940-0b0a-41ed-8ea1-e9ca1e64c467 admin admin] PATH_INFO: `/v3/roles` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:01:54.780105 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37787940-0b0a-41ed-8ea1-e9ca1e64c467 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:01:54.782061 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37787940-0b0a-41ed-8ea1-e9ca1e64c467 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:01:54.793881 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 798/1586] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Aug 30 02:01:54 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 02:01:54.794717 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37787940-0b0a-41ed-8ea1-e9ca1e64c467 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:01:54.805103 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37787940-0b0a-41ed-8ea1-e9ca1e64c467 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:01:54.811840 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd36b863-e6a3-4933-99f0-d71492571be2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:01:54.812655 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd36b863-e6a3-4933-99f0-d71492571be2 admin admin] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:01:54.814795 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd36b863-e6a3-4933-99f0-d71492571be2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:01:54.814795 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd36b863-e6a3-4933-99f0-d71492571be2 admin admin] PATH_INFO: `/v3/users` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:01:54.814795 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd36b863-e6a3-4933-99f0-d71492571be2 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:01:54.816508 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd36b863-e6a3-4933-99f0-d71492571be2 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:01:54.868352 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 789/1587] 10.222.0.27 () {60 vars in 1483 bytes} [Tue Aug 30 02:01:54 2022] PUT /identity/v3/projects/c28e611dd60f47dbb3a396a2b59406c7/users/8019b993d3b74f13ad20564a86f39ab3/roles/1fb81afcb9f245f3b989add94598a063 => generated 0 bytes in 206 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 02:01:54.869323 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3417cb3e-be28-4018-aa37-3aee07a1556a admin admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:01:54.904590 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 799/1588] 10.222.0.27 () {60 vars in 1144 bytes} [Tue Aug 30 02:01:54 2022] POST /identity/v3/users => generated 293 bytes in 110 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 02:01:54.905543 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd36b863-e6a3-4933-99f0-d71492571be2 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:01:54.915668 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd36b863-e6a3-4933-99f0-d71492571be2 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:01:54.921518 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:INSERT=12,keystone:SELECT=88 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:01:54.949772 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3417cb3e-be28-4018-aa37-3aee07a1556a admin admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:01:54.955161 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d523fb4-f5d0-4dd7-b681-36d99b208175 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:01:54.955778 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d523fb4-f5d0-4dd7-b681-36d99b208175 admin admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:01:54.956059 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d523fb4-f5d0-4dd7-b681-36d99b208175 admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:01:54.956330 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d523fb4-f5d0-4dd7-b681-36d99b208175 admin admin] PATH_INFO: `/v3/roles` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:01:54.956977 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d523fb4-f5d0-4dd7-b681-36d99b208175 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:01:54.958989 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d523fb4-f5d0-4dd7-b681-36d99b208175 admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:01:54.974201 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 790/1589] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Aug 30 02:01:54 2022] GET /identity/v3/roles => generated 1420 bytes in 105 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 02:01:54.975057 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d523fb4-f5d0-4dd7-b681-36d99b208175 admin admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:01:54.978603 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ae75bd1-dcf5-4518-a7f0-702b770e2e61 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:01:54.982749 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ae75bd1-dcf5-4518-a7f0-702b770e2e61 admin admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:01:54.982749 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ae75bd1-dcf5-4518-a7f0-702b770e2e61 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:01:54.982749 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ae75bd1-dcf5-4518-a7f0-702b770e2e61 admin admin] PATH_INFO: `/v3/roles` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:01:54.982749 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ae75bd1-dcf5-4518-a7f0-702b770e2e61 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:01:54.985807 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ae75bd1-dcf5-4518-a7f0-702b770e2e61 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:01:54.995077 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d523fb4-f5d0-4dd7-b681-36d99b208175 admin admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:01:55.008032 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 800/1590] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Aug 30 02:01:54 2022] GET /identity/v3/roles => generated 1420 bytes in 99 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 02:01:55.008032 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1f94418-8cdf-471e-af3f-7aa2a2452097 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:01:55.008032 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1f94418-8cdf-471e-af3f-7aa2a2452097 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:01:55.008032 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1f94418-8cdf-471e-af3f-7aa2a2452097 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:01:55.013679 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79b3c4a3-2c3c-4311-a170-66c5b062229d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:01:55.013679 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79b3c4a3-2c3c-4311-a170-66c5b062229d admin admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:01:55.013679 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79b3c4a3-2c3c-4311-a170-66c5b062229d admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:01:55.013679 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79b3c4a3-2c3c-4311-a170-66c5b062229d admin admin] PATH_INFO: `/v3/roles` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:01:55.014335 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79b3c4a3-2c3c-4311-a170-66c5b062229d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:01:55.014335 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79b3c4a3-2c3c-4311-a170-66c5b062229d admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:01:55.025116 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 791/1591] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Aug 30 02:01:54 2022] GET /identity/v3/roles => generated 1420 bytes in 50 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 02:01:55.026011 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79b3c4a3-2c3c-4311-a170-66c5b062229d admin admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:01:55.040496 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79b3c4a3-2c3c-4311-a170-66c5b062229d admin admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:01:55.044870 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-d1f94418-8cdf-471e-af3f-7aa2a2452097 None None] MFA Rules not processed for user `43fc688221d54c3badbb8a6ba3b6d9e3`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:01:55.046447 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04278620-45f6-45b6-9fef-561b59832e0f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:01:55.046983 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-d1f94418-8cdf-471e-af3f-7aa2a2452097 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:01:55.047450 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04278620-45f6-45b6-9fef-561b59832e0f admin admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:01:55.047731 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04278620-45f6-45b6-9fef-561b59832e0f admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:01:55.048029 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04278620-45f6-45b6-9fef-561b59832e0f admin admin] PATH_INFO: `/v3/roles` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:01:55.048676 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04278620-45f6-45b6-9fef-561b59832e0f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:01:55.050306 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04278620-45f6-45b6-9fef-561b59832e0f admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:01:55.063892 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 792/1592] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Aug 30 02:01:55 2022] GET /identity/v3/roles => generated 1420 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 02:01:55.064731 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04278620-45f6-45b6-9fef-561b59832e0f admin admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:01:55.078012 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04278620-45f6-45b6-9fef-561b59832e0f admin admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:01:55.083132 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41d7f163-ae92-4038-bb9f-31c8842e1d1b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:01:55.083798 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41d7f163-ae92-4038-bb9f-31c8842e1d1b admin admin] REQUEST_METHOD: `PUT` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:01:55.084075 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41d7f163-ae92-4038-bb9f-31c8842e1d1b admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:01:55.084346 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41d7f163-ae92-4038-bb9f-31c8842e1d1b admin admin] PATH_INFO: `/v3/projects/bf405bc0c4b5454aa2b868e9cb5fc7e1/users/44f9cfa0ebb84922a9af96ba030ee57a/roles/1fb81afcb9f245f3b989add94598a063` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:01:55.104172 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41d7f163-ae92-4038-bb9f-31c8842e1d1b admin admin] RBAC: Authorizing `identity:create_grant(project_id=bf405bc0c4b5454aa2b868e9cb5fc7e1, user_id=44f9cfa0ebb84922a9af96ba030ee57a, role_id=1fb81afcb9f245f3b989add94598a063)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:01:55.108666 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41d7f163-ae92-4038-bb9f-31c8842e1d1b admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:01:55.135730 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 801/1593] 10.222.0.27 () {58 vars in 958 bytes} [Tue Aug 30 02:01:55 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 132 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 02:01:55.137161 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c275181-0c1c-452c-8876-b46664632d3e None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:01:55.138071 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c275181-0c1c-452c-8876-b46664632d3e None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:01:55.138071 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c275181-0c1c-452c-8876-b46664632d3e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:01:55.170446 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-5c275181-0c1c-452c-8876-b46664632d3e None None] MFA Rules not processed for user `8019b993d3b74f13ad20564a86f39ab3`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:01:55.172022 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-5c275181-0c1c-452c-8876-b46664632d3e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:01:55.174700 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 793/1594] 10.222.0.27 () {60 vars in 1483 bytes} [Tue Aug 30 02:01:55 2022] PUT /identity/v3/projects/bf405bc0c4b5454aa2b868e9cb5fc7e1/users/44f9cfa0ebb84922a9af96ba030ee57a/roles/1fb81afcb9f245f3b989add94598a063 => generated 0 bytes in 110 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 02:01:55.178761 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41d7f163-ae92-4038-bb9f-31c8842e1d1b admin admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:01:55.243185 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41d7f163-ae92-4038-bb9f-31c8842e1d1b admin admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:01:55.249118 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e63cf95-c4f6-4e2a-86f5-671bd0da7699 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:01:55.249771 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e63cf95-c4f6-4e2a-86f5-671bd0da7699 admin admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:01:55.250051 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e63cf95-c4f6-4e2a-86f5-671bd0da7699 admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:01:55.250321 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e63cf95-c4f6-4e2a-86f5-671bd0da7699 admin admin] PATH_INFO: `/v3/roles` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:01:55.250961 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e63cf95-c4f6-4e2a-86f5-671bd0da7699 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:01:55.252560 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e63cf95-c4f6-4e2a-86f5-671bd0da7699 admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:01:55.257982 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 802/1595] 10.222.0.27 () {58 vars in 958 bytes} [Tue Aug 30 02:01:55 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 122 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 02:01:55.259083 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c275181-0c1c-452c-8876-b46664632d3e None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:01:55.269658 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 794/1596] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Aug 30 02:01:55 2022] GET /identity/v3/roles => generated 1420 bytes in 94 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 02:01:55.270401 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c275181-0c1c-452c-8876-b46664632d3e None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:01:55.271123 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e63cf95-c4f6-4e2a-86f5-671bd0da7699 admin admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:01:55.275012 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1a2016b-f743-4281-8ed5-b41adbbb13b5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:01:55.275622 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1a2016b-f743-4281-8ed5-b41adbbb13b5 admin admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:01:55.275899 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1a2016b-f743-4281-8ed5-b41adbbb13b5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:01:55.276194 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1a2016b-f743-4281-8ed5-b41adbbb13b5 admin admin] PATH_INFO: `/v3/roles` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:01:55.277495 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1a2016b-f743-4281-8ed5-b41adbbb13b5 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:01:55.280013 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1a2016b-f743-4281-8ed5-b41adbbb13b5 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:01:55.283432 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e63cf95-c4f6-4e2a-86f5-671bd0da7699 admin admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:01:55.288475 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccd120b7-d8dc-4a4f-8c12-79107b21bd45 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:01:55.289161 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccd120b7-d8dc-4a4f-8c12-79107b21bd45 admin admin] REQUEST_METHOD: `PUT` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:01:55.289443 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccd120b7-d8dc-4a4f-8c12-79107b21bd45 admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:01:55.289741 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccd120b7-d8dc-4a4f-8c12-79107b21bd45 admin admin] PATH_INFO: `/v3/projects/1cbdcc426c3f4d3090919f1a50e3d626/users/bda980735a1f4948ba695e6c2788f3b4/roles/1fb81afcb9f245f3b989add94598a063` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:01:55.292568 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 803/1597] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Aug 30 02:01:55 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 02:01:55.293492 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1a2016b-f743-4281-8ed5-b41adbbb13b5 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:01:55.298705 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:01:55.305265 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1a2016b-f743-4281-8ed5-b41adbbb13b5 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:01:55.310488 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36754f2f-0247-4bb5-96c4-9aefaf866bb2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:01:55.311790 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36754f2f-0247-4bb5-96c4-9aefaf866bb2 admin admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:01:55.312291 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36754f2f-0247-4bb5-96c4-9aefaf866bb2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:01:55.312661 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36754f2f-0247-4bb5-96c4-9aefaf866bb2 admin admin] PATH_INFO: `/v3/roles` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:01:55.312923 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ccd120b7-d8dc-4a4f-8c12-79107b21bd45 admin admin] RBAC: Authorizing `identity:create_grant(project_id=1cbdcc426c3f4d3090919f1a50e3d626, user_id=bda980735a1f4948ba695e6c2788f3b4, role_id=1fb81afcb9f245f3b989add94598a063)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:01:55.313395 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36754f2f-0247-4bb5-96c4-9aefaf866bb2 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:01:55.315119 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36754f2f-0247-4bb5-96c4-9aefaf866bb2 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:01:55.315781 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ccd120b7-d8dc-4a4f-8c12-79107b21bd45 admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:01:55.329239 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 804/1598] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Aug 30 02:01:55 2022] GET /identity/v3/roles => generated 1420 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 02:01:55.330306 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36754f2f-0247-4bb5-96c4-9aefaf866bb2 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:01:55.336636 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:01:55.344479 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36754f2f-0247-4bb5-96c4-9aefaf866bb2 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:01:55.351148 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b9febee-417f-4628-8545-09ec66e411d9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:01:55.352022 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b9febee-417f-4628-8545-09ec66e411d9 admin admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:01:55.352437 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b9febee-417f-4628-8545-09ec66e411d9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:01:55.352833 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b9febee-417f-4628-8545-09ec66e411d9 admin admin] PATH_INFO: `/v3/roles` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:01:55.353905 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b9febee-417f-4628-8545-09ec66e411d9 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:01:55.356391 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b9febee-417f-4628-8545-09ec66e411d9 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:01:55.372607 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 805/1599] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Aug 30 02:01:55 2022] GET /identity/v3/roles => generated 1420 bytes in 43 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 02:01:55.374081 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b9febee-417f-4628-8545-09ec66e411d9 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:01:55.384775 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b9febee-417f-4628-8545-09ec66e411d9 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:01:55.389805 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec4d59b8-ef94-4373-841d-666f8256d2b5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:01:55.390414 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec4d59b8-ef94-4373-841d-666f8256d2b5 admin admin] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:01:55.390685 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec4d59b8-ef94-4373-841d-666f8256d2b5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:01:55.390952 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec4d59b8-ef94-4373-841d-666f8256d2b5 admin admin] PATH_INFO: `/v3/projects` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:01:55.391577 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec4d59b8-ef94-4373-841d-666f8256d2b5 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:01:55.393670 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec4d59b8-ef94-4373-841d-666f8256d2b5 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:01:55.402790 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 795/1600] 10.222.0.27 () {60 vars in 1483 bytes} [Tue Aug 30 02:01:55 2022] PUT /identity/v3/projects/1cbdcc426c3f4d3090919f1a50e3d626/users/bda980735a1f4948ba695e6c2788f3b4/roles/1fb81afcb9f245f3b989add94598a063 => generated 0 bytes in 132 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 02:01:55.404187 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2eb32d4b-d334-48c5-bd1f-27bf2e7ba18a None None] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:01:55.404631 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2eb32d4b-d334-48c5-bd1f-27bf2e7ba18a None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:01:55.405086 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2eb32d4b-d334-48c5-bd1f-27bf2e7ba18a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:01:55.431374 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.auth.core [None req-2eb32d4b-d334-48c5-bd1f-27bf2e7ba18a None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81578) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:01:55.432652 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-2eb32d4b-d334-48c5-bd1f-27bf2e7ba18a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:01:55.466923 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 806/1601] 10.222.0.27 () {60 vars in 1153 bytes} [Tue Aug 30 02:01:55 2022] POST /identity/v3/projects => generated 379 bytes in 94 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 02:01:55.469641 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec4d59b8-ef94-4373-841d-666f8256d2b5 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:01:55.504396 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 796/1602] 10.222.0.27 () {58 vars in 958 bytes} [Tue Aug 30 02:01:55 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 02:01:55.505794 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8a23fef-431c-40b5-b1ab-db9b54964001 None None] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:01:55.506146 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8a23fef-431c-40b5-b1ab-db9b54964001 None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:01:55.506544 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8a23fef-431c-40b5-b1ab-db9b54964001 None None] PATH_INFO: `/` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:01:55.507904 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 797/1603] 10.222.0.27 () {54 vars in 962 bytes} [Tue Aug 30 02:01:55 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 02:01:55.508830 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8a23fef-431c-40b5-b1ab-db9b54964001 None None] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:01:55.539364 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec4d59b8-ef94-4373-841d-666f8256d2b5 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:01:55.544233 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35e92cb0-962c-470c-beab-bad45666c8a9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:01:55.544831 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35e92cb0-962c-470c-beab-bad45666c8a9 admin admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:01:55.545658 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35e92cb0-962c-470c-beab-bad45666c8a9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:01:55.545937 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35e92cb0-962c-470c-beab-bad45666c8a9 admin admin] PATH_INFO: `/v3/roles` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:01:55.546580 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35e92cb0-962c-470c-beab-bad45666c8a9 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:01:55.548221 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35e92cb0-962c-470c-beab-bad45666c8a9 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:01:55.558987 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 807/1604] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Aug 30 02:01:55 2022] GET /identity/v3/roles => generated 1420 bytes in 91 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 02:01:55.560098 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35e92cb0-962c-470c-beab-bad45666c8a9 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:01:55.570245 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35e92cb0-962c-470c-beab-bad45666c8a9 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:01:55.574855 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fa6e5e1-f98f-4a00-ba85-181d1f891b1b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:01:55.575370 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8a23fef-431c-40b5-b1ab-db9b54964001 None None] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:01:55.575480 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fa6e5e1-f98f-4a00-ba85-181d1f891b1b admin admin] REQUEST_METHOD: `PUT` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:01:55.575762 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fa6e5e1-f98f-4a00-ba85-181d1f891b1b admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:01:55.576033 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fa6e5e1-f98f-4a00-ba85-181d1f891b1b admin admin] PATH_INFO: `/v3/projects/a06899b187644ff4a2c5a5a38d0ab751/users/26c77ddb8a184d1ea623f379d492768b/roles/1fb81afcb9f245f3b989add94598a063` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:01:55.579856 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa86bacf-a721-42ba-967f-a2d349167208 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:01:55.580512 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa86bacf-a721-42ba-967f-a2d349167208 admin admin] REQUEST_METHOD: `PUT` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:01:55.580790 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa86bacf-a721-42ba-967f-a2d349167208 admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:01:55.581058 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa86bacf-a721-42ba-967f-a2d349167208 admin admin] PATH_INFO: `/v3/projects/e7f4b9bbe0b640688754d04484900cca/users/5e7584fe83684b01baa31ab67edac026/roles/1fb81afcb9f245f3b989add94598a063` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:01:55.590785 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fa6e5e1-f98f-4a00-ba85-181d1f891b1b admin admin] RBAC: Authorizing `identity:create_grant(project_id=a06899b187644ff4a2c5a5a38d0ab751, user_id=26c77ddb8a184d1ea623f379d492768b, role_id=1fb81afcb9f245f3b989add94598a063)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:01:55.592009 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: WARNING py.warnings [None req-5fa6e5e1-f98f-4a00-ba85-181d1f891b1b admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_grant": "(role:admin and system_scope:all) or ((role:admin and domain_id:%(target.user.domain_id)s and domain_id:%(target.project.domain_id)s) or (role:admin and domain_id:%(target.user.domain_id)s and domain_id:%(target.domain.id)s) or (role:admin and domain_id:%(target.group.domain_id)s and domain_id:%(target.project.domain_id)s) or (role:admin and domain_id:%(target.group.domain_id)s and domain_id:%(target.domain.id)s)) and (domain_id:%(target.role.domain_id)s or None:%(target.role.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Aug 30 02:01:55.592009 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: warnings.warn(msg) Aug 30 02:01:55.592009 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: Aug 30 02:01:55.594228 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fa6e5e1-f98f-4a00-ba85-181d1f891b1b admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:01:55.595608 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa86bacf-a721-42ba-967f-a2d349167208 admin admin] RBAC: Authorizing `identity:create_grant(project_id=e7f4b9bbe0b640688754d04484900cca, user_id=5e7584fe83684b01baa31ab67edac026, role_id=1fb81afcb9f245f3b989add94598a063)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:01:55.598297 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa86bacf-a721-42ba-967f-a2d349167208 admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:01:55.678620 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 808/1605] 10.222.0.27 () {60 vars in 1483 bytes} [Tue Aug 30 02:01:55 2022] PUT /identity/v3/projects/a06899b187644ff4a2c5a5a38d0ab751/users/26c77ddb8a184d1ea623f379d492768b/roles/1fb81afcb9f245f3b989add94598a063 => generated 0 bytes in 119 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 02:01:55.679737 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fa6e5e1-f98f-4a00-ba85-181d1f891b1b admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:01:55.689947 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 798/1606] 10.222.0.27 () {60 vars in 1483 bytes} [Tue Aug 30 02:01:55 2022] PUT /identity/v3/projects/e7f4b9bbe0b640688754d04484900cca/users/5e7584fe83684b01baa31ab67edac026/roles/1fb81afcb9f245f3b989add94598a063 => generated 0 bytes in 181 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 02:01:55.691763 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c136312-dce7-4fbd-8cca-a7004ed278ac None None] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:01:55.692106 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c136312-dce7-4fbd-8cca-a7004ed278ac None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:01:55.692635 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c136312-dce7-4fbd-8cca-a7004ed278ac None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:01:55.713232 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.auth.core [None req-7c136312-dce7-4fbd-8cca-a7004ed278ac None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81578) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:01:55.714468 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-7c136312-dce7-4fbd-8cca-a7004ed278ac None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:01:55.755538 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fa6e5e1-f98f-4a00-ba85-181d1f891b1b admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:01:55.776353 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 799/1607] 10.222.0.27 () {58 vars in 958 bytes} [Tue Aug 30 02:01:55 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 02:01:55.777187 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c136312-dce7-4fbd-8cca-a7004ed278ac None None] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:01:55.787176 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c136312-dce7-4fbd-8cca-a7004ed278ac None None] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:01:55.791814 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c326994c-00f3-4562-822d-ef820dfbf891 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:01:55.792412 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c326994c-00f3-4562-822d-ef820dfbf891 admin admin] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:01:55.792691 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c326994c-00f3-4562-822d-ef820dfbf891 admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:01:55.792963 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c326994c-00f3-4562-822d-ef820dfbf891 admin admin] PATH_INFO: `/v3/users` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:01:55.793624 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c326994c-00f3-4562-822d-ef820dfbf891 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:01:55.796226 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c326994c-00f3-4562-822d-ef820dfbf891 admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:01:55.801875 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:01:55.805483 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:01:55.813902 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f798f5b-5950-4f37-8a7d-279f0ad53b9c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:01:55.814608 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f798f5b-5950-4f37-8a7d-279f0ad53b9c admin admin] REQUEST_METHOD: `PUT` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:01:55.814887 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f798f5b-5950-4f37-8a7d-279f0ad53b9c admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:01:55.815162 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f798f5b-5950-4f37-8a7d-279f0ad53b9c admin admin] PATH_INFO: `/v3/projects/bacfc2716dad4ad991b56e781f2d05ef/users/9bb1df3fe2214e9da8e72b1047d569f2/roles/1fb81afcb9f245f3b989add94598a063` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:01:55.829793 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f798f5b-5950-4f37-8a7d-279f0ad53b9c admin admin] RBAC: Authorizing `identity:create_grant(project_id=bacfc2716dad4ad991b56e781f2d05ef, user_id=9bb1df3fe2214e9da8e72b1047d569f2, role_id=1fb81afcb9f245f3b989add94598a063)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:01:55.832859 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f798f5b-5950-4f37-8a7d-279f0ad53b9c admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:01:55.924583 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 800/1608] 10.222.0.27 () {60 vars in 1144 bytes} [Tue Aug 30 02:01:55 2022] POST /identity/v3/users => generated 296 bytes in 145 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 02:01:55.924583 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a43a41f-a512-4de3-b4c2-712a29e3f218 None None] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:01:55.924583 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a43a41f-a512-4de3-b4c2-712a29e3f218 None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:01:55.924583 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a43a41f-a512-4de3-b4c2-712a29e3f218 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:01:55.944737 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 809/1609] 10.222.0.27 () {60 vars in 1483 bytes} [Tue Aug 30 02:01:55 2022] PUT /identity/v3/projects/bacfc2716dad4ad991b56e781f2d05ef/users/9bb1df3fe2214e9da8e72b1047d569f2/roles/1fb81afcb9f245f3b989add94598a063 => generated 0 bytes in 265 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 02:01:55.945103 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f798f5b-5950-4f37-8a7d-279f0ad53b9c admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:01:55.962296 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.auth.core [None req-5a43a41f-a512-4de3-b4c2-712a29e3f218 None None] MFA Rules not processed for user `d64f2e71361644589bec8a06e6eceed1`. Rule list: `[]` (Enabled: `True`). {{(pid=81578) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:01:55.963628 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-5a43a41f-a512-4de3-b4c2-712a29e3f218 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:01:56.015425 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f798f5b-5950-4f37-8a7d-279f0ad53b9c admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:01:56.019279 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bb80454-76d6-4d44-a52f-26a79ba29e5f service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f3ea1580c56f4814a92bbb2acf542905', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4562f3951fbe465280073e60a11e44e2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:01:56.019912 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bb80454-76d6-4d44-a52f-26a79ba29e5f service neutron] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:01:56.020190 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bb80454-76d6-4d44-a52f-26a79ba29e5f service neutron] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:01:56.020460 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bb80454-76d6-4d44-a52f-26a79ba29e5f service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:01:56.033201 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 801/1610] 10.222.0.27 () {58 vars in 1059 bytes} [Tue Aug 30 02:01:55 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 111 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 02:01:56.034095 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a43a41f-a512-4de3-b4c2-712a29e3f218 None None] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:01:56.086725 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5bb80454-76d6-4d44-a52f-26a79ba29e5f service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:01:56.088166 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5bb80454-76d6-4d44-a52f-26a79ba29e5f service neutron] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:01:56.098913 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 810/1611] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Aug 30 02:01:55 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 154 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 02:01:56.099216 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a43a41f-a512-4de3-b4c2-712a29e3f218 None None] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:01:56.100218 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0bc1fa8-8488-425d-84b1-36f6f5cfc1a3 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:01:56.100454 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0bc1fa8-8488-425d-84b1-36f6f5cfc1a3 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:01:56.100727 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0bc1fa8-8488-425d-84b1-36f6f5cfc1a3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:01:56.103868 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5331de4c-a30e-4108-9cc9-b0ca3c6f6785 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:01:56.104532 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5331de4c-a30e-4108-9cc9-b0ca3c6f6785 admin admin] REQUEST_METHOD: `PUT` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:01:56.104894 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5331de4c-a30e-4108-9cc9-b0ca3c6f6785 admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:01:56.105091 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5331de4c-a30e-4108-9cc9-b0ca3c6f6785 admin admin] PATH_INFO: `/v3/projects/c57bc7a319f14bf393e6249b2cb59c69/users/7af1c1123b2e458b84248865287bd00e/roles/1fb81afcb9f245f3b989add94598a063` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:01:56.121306 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5331de4c-a30e-4108-9cc9-b0ca3c6f6785 admin admin] RBAC: Authorizing `identity:create_grant(project_id=c57bc7a319f14bf393e6249b2cb59c69, user_id=7af1c1123b2e458b84248865287bd00e, role_id=1fb81afcb9f245f3b989add94598a063)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:01:56.124118 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5331de4c-a30e-4108-9cc9-b0ca3c6f6785 admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:01:56.131244 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-a0bc1fa8-8488-425d-84b1-36f6f5cfc1a3 None None] MFA Rules not processed for user `26c77ddb8a184d1ea623f379d492768b`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:01:56.132445 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-a0bc1fa8-8488-425d-84b1-36f6f5cfc1a3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:01:56.179203 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 802/1612] 10.222.0.27 () {60 vars in 1483 bytes} [Tue Aug 30 02:01:56 2022] PUT /identity/v3/projects/c57bc7a319f14bf393e6249b2cb59c69/users/7af1c1123b2e458b84248865287bd00e/roles/1fb81afcb9f245f3b989add94598a063 => generated 0 bytes in 145 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 02:01:56.180488 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa8f3c47-5a8b-47b4-8bc2-346ac5b166d1 None None] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:01:56.180792 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa8f3c47-5a8b-47b4-8bc2-346ac5b166d1 None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:01:56.181053 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa8f3c47-5a8b-47b4-8bc2-346ac5b166d1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:01:56.197443 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 811/1613] 10.222.0.27 () {58 vars in 958 bytes} [Tue Aug 30 02:01:56 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 02:01:56.198434 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0bc1fa8-8488-425d-84b1-36f6f5cfc1a3 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:01:56.200064 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.auth.core [None req-fa8f3c47-5a8b-47b4-8bc2-346ac5b166d1 None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81578) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:01:56.201824 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-fa8f3c47-5a8b-47b4-8bc2-346ac5b166d1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:01:56.208616 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0bc1fa8-8488-425d-84b1-36f6f5cfc1a3 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:01:56.219801 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:01:56.260243 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:INSERT=3,keystone:SELECT=97 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:01:56.262584 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99f7e769-da1f-4035-a7e2-abb6ade186da service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f3ea1580c56f4814a92bbb2acf542905', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4562f3951fbe465280073e60a11e44e2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:01:56.263336 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99f7e769-da1f-4035-a7e2-abb6ade186da service neutron] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:01:56.263625 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99f7e769-da1f-4035-a7e2-abb6ade186da service neutron] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:01:56.264475 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99f7e769-da1f-4035-a7e2-abb6ade186da service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:01:56.280427 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 803/1614] 10.222.0.27 () {58 vars in 958 bytes} [Tue Aug 30 02:01:56 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 02:01:56.280427 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa8f3c47-5a8b-47b4-8bc2-346ac5b166d1 None None] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:01:56.293381 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa8f3c47-5a8b-47b4-8bc2-346ac5b166d1 None None] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:01:56.298782 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e4a7053-3c2d-40cc-bc6a-31772c7d3b3d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:01:56.299396 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e4a7053-3c2d-40cc-bc6a-31772c7d3b3d admin admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:01:56.299671 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e4a7053-3c2d-40cc-bc6a-31772c7d3b3d admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:01:56.300064 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e4a7053-3c2d-40cc-bc6a-31772c7d3b3d admin admin] PATH_INFO: `/v3/roles` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:01:56.300652 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e4a7053-3c2d-40cc-bc6a-31772c7d3b3d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:01:56.302303 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e4a7053-3c2d-40cc-bc6a-31772c7d3b3d admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:01:56.312584 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 804/1615] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Aug 30 02:01:56 2022] GET /identity/v3/roles => generated 1420 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 02:01:56.313987 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76f0451f-1c48-480a-8016-99bcebfd5726 None None] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:01:56.314271 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76f0451f-1c48-480a-8016-99bcebfd5726 None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:01:56.314541 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76f0451f-1c48-480a-8016-99bcebfd5726 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:01:56.336437 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99f7e769-da1f-4035-a7e2-abb6ade186da service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:01:56.337977 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99f7e769-da1f-4035-a7e2-abb6ade186da service neutron] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:01:56.343024 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.auth.core [None req-76f0451f-1c48-480a-8016-99bcebfd5726 None None] MFA Rules not processed for user `9bb1df3fe2214e9da8e72b1047d569f2`. Rule list: `[]` (Enabled: `True`). {{(pid=81578) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:01:56.344516 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-76f0451f-1c48-480a-8016-99bcebfd5726 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:01:56.348523 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 812/1616] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Aug 30 02:01:56 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 151 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 02:01:56.349324 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99f7e769-da1f-4035-a7e2-abb6ade186da service neutron] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:01:56.419930 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 805/1617] 10.222.0.27 () {58 vars in 958 bytes} [Tue Aug 30 02:01:56 2022] POST /identity/v3/auth/tokens => generated 3248 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 02:01:56.426655 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0f02d52-3afc-4e38-92e3-20532e4c3a29 None None] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:01:56.426655 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0f02d52-3afc-4e38-92e3-20532e4c3a29 None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:01:56.426655 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0f02d52-3afc-4e38-92e3-20532e4c3a29 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:01:56.432105 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99f7e769-da1f-4035-a7e2-abb6ade186da service neutron] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:01:56.438899 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3848bf9d-4148-457f-84d7-42195e2ff35d service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd64f2e71361644589bec8a06e6eceed1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4562f3951fbe465280073e60a11e44e2', 'project_domain_id': 'default', 'roles': ['member', 'service', 'admin', 'ResellerAdmin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:01:56.439757 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3848bf9d-4148-457f-84d7-42195e2ff35d service nova] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:01:56.439943 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3848bf9d-4148-457f-84d7-42195e2ff35d service nova] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:01:56.440179 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3848bf9d-4148-457f-84d7-42195e2ff35d service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:01:56.442324 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.auth.core [None req-f0f02d52-3afc-4e38-92e3-20532e4c3a29 None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81578) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:01:56.443855 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-f0f02d52-3afc-4e38-92e3-20532e4c3a29 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:01:56.461162 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 806/1618] 10.222.0.27 () {58 vars in 958 bytes} [Tue Aug 30 02:01:56 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 38 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 02:01:56.461162 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0f02d52-3afc-4e38-92e3-20532e4c3a29 None None] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:01:56.471628 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0f02d52-3afc-4e38-92e3-20532e4c3a29 None None] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:01:56.478384 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff67d3d3-428c-4f0d-8f33-456ae610c59a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:01:56.478384 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff67d3d3-428c-4f0d-8f33-456ae610c59a admin admin] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:01:56.478384 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff67d3d3-428c-4f0d-8f33-456ae610c59a admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:01:56.478384 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff67d3d3-428c-4f0d-8f33-456ae610c59a admin admin] PATH_INFO: `/v3/projects` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:01:56.478785 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff67d3d3-428c-4f0d-8f33-456ae610c59a admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:01:56.480032 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff67d3d3-428c-4f0d-8f33-456ae610c59a admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:01:56.514966 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3848bf9d-4148-457f-84d7-42195e2ff35d service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:01:56.516616 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3848bf9d-4148-457f-84d7-42195e2ff35d service nova] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:01:56.520318 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:01:56.526114 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 813/1619] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Aug 30 02:01:56 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 177 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 02:01:56.527011 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3848bf9d-4148-457f-84d7-42195e2ff35d service nova] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:01:56.528389 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-3848bf9d-4148-457f-84d7-42195e2ff35d service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:01:56.541268 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3848bf9d-4148-457f-84d7-42195e2ff35d service nova] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:01:56.558329 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e820fbf4-1098-43f8-b582-9ac6c788b1cf service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f3ea1580c56f4814a92bbb2acf542905', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4562f3951fbe465280073e60a11e44e2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:01:56.559206 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e820fbf4-1098-43f8-b582-9ac6c788b1cf service neutron] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:01:56.559699 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e820fbf4-1098-43f8-b582-9ac6c788b1cf service neutron] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:01:56.560238 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e820fbf4-1098-43f8-b582-9ac6c788b1cf service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:01:56.586204 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e820fbf4-1098-43f8-b582-9ac6c788b1cf service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:01:56.586204 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e820fbf4-1098-43f8-b582-9ac6c788b1cf service neutron] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:01:56.588220 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 807/1620] 10.222.0.27 () {60 vars in 1153 bytes} [Tue Aug 30 02:01:56 2022] POST /identity/v3/projects => generated 387 bytes in 129 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 02:01:56.589212 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff67d3d3-428c-4f0d-8f33-456ae610c59a admin admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:01:56.601963 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 814/1621] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Aug 30 02:01:56 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 75 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 02:01:56.604287 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4e6ff6f-10a8-44da-a298-6def8735a056 None None] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:01:56.604727 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4e6ff6f-10a8-44da-a298-6def8735a056 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:01:56.605102 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4e6ff6f-10a8-44da-a298-6def8735a056 None None] PATH_INFO: `/` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:01:56.607127 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 815/1622] 10.222.0.27 () {54 vars in 962 bytes} [Tue Aug 30 02:01:56 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 02:01:56.610044 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4e6ff6f-10a8-44da-a298-6def8735a056 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:01:56.677354 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4e6ff6f-10a8-44da-a298-6def8735a056 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:01:56.681751 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4d120dd-9969-4c2c-a358-eba661155ad4 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f3ea1580c56f4814a92bbb2acf542905', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4562f3951fbe465280073e60a11e44e2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:01:56.682616 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4d120dd-9969-4c2c-a358-eba661155ad4 service neutron] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:01:56.682959 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4d120dd-9969-4c2c-a358-eba661155ad4 service neutron] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:01:56.683322 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4d120dd-9969-4c2c-a358-eba661155ad4 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:01:56.683550 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff67d3d3-428c-4f0d-8f33-456ae610c59a admin admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:01:56.690005 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fdaf31a-0bcb-42a2-ac0d-d29d60d2645d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:01:56.690741 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fdaf31a-0bcb-42a2-ac0d-d29d60d2645d admin admin] REQUEST_METHOD: `PUT` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:01:56.691443 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fdaf31a-0bcb-42a2-ac0d-d29d60d2645d admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:01:56.691751 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fdaf31a-0bcb-42a2-ac0d-d29d60d2645d admin admin] PATH_INFO: `/v3/projects/0da6a09895c54fe3a759d06eb7773159/users/e8898371204546a89e21b6e34aaa04eb/roles/ead5fac86a0747ea8823a82efcea8ed7` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:01:56.699095 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4d120dd-9969-4c2c-a358-eba661155ad4 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:01:56.702113 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4d120dd-9969-4c2c-a358-eba661155ad4 service neutron] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:01:56.713928 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 816/1623] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Aug 30 02:01:56 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 106 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 02:01:56.714832 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4d120dd-9969-4c2c-a358-eba661155ad4 service neutron] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:01:56.725574 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fdaf31a-0bcb-42a2-ac0d-d29d60d2645d admin admin] RBAC: Authorizing `identity:create_grant(project_id=0da6a09895c54fe3a759d06eb7773159, user_id=e8898371204546a89e21b6e34aaa04eb, role_id=ead5fac86a0747ea8823a82efcea8ed7)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:01:56.727301 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4d120dd-9969-4c2c-a358-eba661155ad4 service neutron] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:01:56.728417 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fdaf31a-0bcb-42a2-ac0d-d29d60d2645d admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:01:56.733112 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f100d91d-c4bc-4a6e-a4fd-117635dcacc8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:01:56.733943 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f100d91d-c4bc-4a6e-a4fd-117635dcacc8 admin admin] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:01:56.734314 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f100d91d-c4bc-4a6e-a4fd-117635dcacc8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:01:56.734671 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f100d91d-c4bc-4a6e-a4fd-117635dcacc8 admin admin] PATH_INFO: `/v3/users` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:01:56.735403 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f100d91d-c4bc-4a6e-a4fd-117635dcacc8 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:01:56.737543 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f100d91d-c4bc-4a6e-a4fd-117635dcacc8 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:01:56.792215 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 808/1624] 10.222.0.27 () {60 vars in 1483 bytes} [Tue Aug 30 02:01:56 2022] PUT /identity/v3/projects/0da6a09895c54fe3a759d06eb7773159/users/e8898371204546a89e21b6e34aaa04eb/roles/ead5fac86a0747ea8823a82efcea8ed7 => generated 0 bytes in 203 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 02:01:56.793869 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36b419bf-4225-4bab-a72d-c0df7a7603ef None None] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:01:56.794207 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36b419bf-4225-4bab-a72d-c0df7a7603ef None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:01:56.794533 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36b419bf-4225-4bab-a72d-c0df7a7603ef None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:01:56.821038 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.auth.core [None req-36b419bf-4225-4bab-a72d-c0df7a7603ef None None] MFA Rules not processed for user `d64f2e71361644589bec8a06e6eceed1`. Rule list: `[]` (Enabled: `True`). {{(pid=81578) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:01:56.822248 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-36b419bf-4225-4bab-a72d-c0df7a7603ef None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:01:56.822885 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 817/1625] 10.222.0.27 () {60 vars in 1144 bytes} [Tue Aug 30 02:01:56 2022] POST /identity/v3/users => generated 300 bytes in 108 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 02:01:56.823795 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f100d91d-c4bc-4a6e-a4fd-117635dcacc8 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:01:56.856480 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:01:56.862411 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f100d91d-c4bc-4a6e-a4fd-117635dcacc8 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:01:56.912347 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 809/1626] 10.222.0.27 () {58 vars in 1059 bytes} [Tue Aug 30 02:01:56 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 119 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 02:01:56.913217 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36b419bf-4225-4bab-a72d-c0df7a7603ef None None] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:01:57.019799 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36b419bf-4225-4bab-a72d-c0df7a7603ef None None] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:01:57.027601 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77b3cd2a-b687-4647-b307-e7e57400a651 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:01:57.030713 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77b3cd2a-b687-4647-b307-e7e57400a651 admin admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:01:57.031011 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77b3cd2a-b687-4647-b307-e7e57400a651 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:01:57.031251 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77b3cd2a-b687-4647-b307-e7e57400a651 admin admin] PATH_INFO: `/v3/roles` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:01:57.033819 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77b3cd2a-b687-4647-b307-e7e57400a651 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:01:57.037367 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77b3cd2a-b687-4647-b307-e7e57400a651 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:01:57.039064 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7f9e3ac-ea32-4604-a9fa-75105da29503 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:01:57.039693 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7f9e3ac-ea32-4604-a9fa-75105da29503 admin admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:01:57.042435 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7f9e3ac-ea32-4604-a9fa-75105da29503 admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:01:57.042435 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7f9e3ac-ea32-4604-a9fa-75105da29503 admin admin] PATH_INFO: `/v3/roles` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:01:57.042435 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7f9e3ac-ea32-4604-a9fa-75105da29503 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:01:57.044963 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7f9e3ac-ea32-4604-a9fa-75105da29503 admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:01:57.059247 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 810/1627] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Aug 30 02:01:56 2022] GET /identity/v3/roles => generated 1420 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 02:01:57.065645 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7f9e3ac-ea32-4604-a9fa-75105da29503 admin admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:01:57.065780 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 818/1628] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Aug 30 02:01:56 2022] GET /identity/v3/roles => generated 1420 bytes in 240 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 02:01:57.065780 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a569a82-7e6a-4a14-9757-0788d2f0afd5 None None] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:01:57.065780 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a569a82-7e6a-4a14-9757-0788d2f0afd5 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:01:57.065780 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a569a82-7e6a-4a14-9757-0788d2f0afd5 None None] PATH_INFO: `/` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:01:57.067356 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 819/1629] 10.222.0.27 () {54 vars in 912 bytes} [Tue Aug 30 02:01:57 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 02:01:57.068527 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a569a82-7e6a-4a14-9757-0788d2f0afd5 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:01:57.084113 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a569a82-7e6a-4a14-9757-0788d2f0afd5 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:01:57.090343 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02761a23-0835-43fa-a4ab-520875eb852d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:01:57.091339 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02761a23-0835-43fa-a4ab-520875eb852d admin admin] REQUEST_METHOD: `PUT` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:01:57.096789 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02761a23-0835-43fa-a4ab-520875eb852d admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:01:57.096789 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02761a23-0835-43fa-a4ab-520875eb852d admin admin] PATH_INFO: `/v3/projects/77275929094b47d291adf3fe1776f385/users/4cce43a9515d48f89a629be170d4da2d/roles/ead5fac86a0747ea8823a82efcea8ed7` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:01:57.108048 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7f9e3ac-ea32-4604-a9fa-75105da29503 admin admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:01:57.116191 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02761a23-0835-43fa-a4ab-520875eb852d admin admin] RBAC: Authorizing `identity:create_grant(project_id=77275929094b47d291adf3fe1776f385, user_id=4cce43a9515d48f89a629be170d4da2d, role_id=ead5fac86a0747ea8823a82efcea8ed7)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:01:57.116997 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be8bbaed-3411-4ba8-bf97-02c313788027 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd64f2e71361644589bec8a06e6eceed1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4562f3951fbe465280073e60a11e44e2', 'project_domain_id': 'default', 'roles': ['member', 'service', 'admin', 'ResellerAdmin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:01:57.117649 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be8bbaed-3411-4ba8-bf97-02c313788027 service nova] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:01:57.117931 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be8bbaed-3411-4ba8-bf97-02c313788027 service nova] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:01:57.118201 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be8bbaed-3411-4ba8-bf97-02c313788027 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:01:57.119425 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02761a23-0835-43fa-a4ab-520875eb852d admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:01:57.185540 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be8bbaed-3411-4ba8-bf97-02c313788027 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:01:57.185540 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be8bbaed-3411-4ba8-bf97-02c313788027 service nova] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:01:57.195901 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 811/1630] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Aug 30 02:01:57 2022] GET /identity/v3/auth/tokens => generated 3248 bytes in 136 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 02:01:57.196717 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be8bbaed-3411-4ba8-bf97-02c313788027 service nova] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:01:57.218244 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 820/1631] 10.222.0.27 () {60 vars in 1483 bytes} [Tue Aug 30 02:01:57 2022] PUT /identity/v3/projects/77275929094b47d291adf3fe1776f385/users/4cce43a9515d48f89a629be170d4da2d/roles/ead5fac86a0747ea8823a82efcea8ed7 => generated 0 bytes in 146 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 02:01:57.220953 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e86b90d-774b-49ce-b577-214122d3bbbc None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:01:57.220953 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e86b90d-774b-49ce-b577-214122d3bbbc None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:01:57.221166 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e86b90d-774b-49ce-b577-214122d3bbbc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:01:57.228688 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be8bbaed-3411-4ba8-bf97-02c313788027 service nova] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:01:57.266124 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-1e86b90d-774b-49ce-b577-214122d3bbbc None None] MFA Rules not processed for user `d64f2e71361644589bec8a06e6eceed1`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:01:57.271160 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-1e86b90d-774b-49ce-b577-214122d3bbbc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:01:57.324489 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:01:57.339153 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26989543-2a94-4d3e-ae10-5cc9f1ac3992 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:01:57.339831 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26989543-2a94-4d3e-ae10-5cc9f1ac3992 admin admin] REQUEST_METHOD: `PUT` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:01:57.340164 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26989543-2a94-4d3e-ae10-5cc9f1ac3992 admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:01:57.340472 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26989543-2a94-4d3e-ae10-5cc9f1ac3992 admin admin] PATH_INFO: `/v3/projects/0da6a09895c54fe3a759d06eb7773159/users/e8898371204546a89e21b6e34aaa04eb/roles/ead5fac86a0747ea8823a82efcea8ed7` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:01:57.347118 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26989543-2a94-4d3e-ae10-5cc9f1ac3992 admin admin] RBAC: Authorizing `identity:create_grant(project_id=0da6a09895c54fe3a759d06eb7773159, user_id=e8898371204546a89e21b6e34aaa04eb, role_id=ead5fac86a0747ea8823a82efcea8ed7)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:01:57.352962 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26989543-2a94-4d3e-ae10-5cc9f1ac3992 admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:01:57.359248 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:01:57.387628 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 812/1632] 10.222.0.27 () {60 vars in 1483 bytes} [Tue Aug 30 02:01:57 2022] PUT /identity/v3/projects/0da6a09895c54fe3a759d06eb7773159/users/e8898371204546a89e21b6e34aaa04eb/roles/ead5fac86a0747ea8823a82efcea8ed7 => generated 0 bytes in 191 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 02:01:57.388567 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26989543-2a94-4d3e-ae10-5cc9f1ac3992 admin admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:01:57.428541 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 821/1633] 10.222.0.27 () {58 vars in 1009 bytes} [Tue Aug 30 02:01:57 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 210 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 02:01:57.433200 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0636fce8-e360-4050-904d-30b992e2bc94 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:01:57.433200 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0636fce8-e360-4050-904d-30b992e2bc94 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:01:57.433200 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0636fce8-e360-4050-904d-30b992e2bc94 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:01:57.483683 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-0636fce8-e360-4050-904d-30b992e2bc94 None None] MFA Rules not processed for user `e8898371204546a89e21b6e34aaa04eb`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:01:57.485041 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-0636fce8-e360-4050-904d-30b992e2bc94 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:01:57.496745 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26989543-2a94-4d3e-ae10-5cc9f1ac3992 admin admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:01:57.503770 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8afe754-d638-4151-b3ce-f8e42ab98da4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:01:57.504986 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8afe754-d638-4151-b3ce-f8e42ab98da4 admin admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:01:57.505410 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8afe754-d638-4151-b3ce-f8e42ab98da4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:01:57.505880 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8afe754-d638-4151-b3ce-f8e42ab98da4 admin admin] PATH_INFO: `/v3/roles` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:01:57.506849 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8afe754-d638-4151-b3ce-f8e42ab98da4 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:01:57.508853 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8afe754-d638-4151-b3ce-f8e42ab98da4 admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:01:57.550962 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 813/1634] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Aug 30 02:01:57 2022] GET /identity/v3/roles => generated 1420 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 02:01:57.556183 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8afe754-d638-4151-b3ce-f8e42ab98da4 admin admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:01:57.605090 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 822/1635] 10.222.0.27 () {58 vars in 958 bytes} [Tue Aug 30 02:01:57 2022] POST /identity/v3/auth/tokens => generated 3323 bytes in 176 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 02:01:57.606638 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0636fce8-e360-4050-904d-30b992e2bc94 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:01:57.619756 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0636fce8-e360-4050-904d-30b992e2bc94 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:01:57.624968 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7d23529-daa3-4a50-9b1d-62431aec9488 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:01:57.625752 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7d23529-daa3-4a50-9b1d-62431aec9488 admin admin] REQUEST_METHOD: `PUT` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:01:57.627394 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7d23529-daa3-4a50-9b1d-62431aec9488 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:01:57.627394 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7d23529-daa3-4a50-9b1d-62431aec9488 admin admin] PATH_INFO: `/v3/projects/77275929094b47d291adf3fe1776f385/users/4cce43a9515d48f89a629be170d4da2d/roles/ead5fac86a0747ea8823a82efcea8ed7` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:01:57.629931 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7d23529-daa3-4a50-9b1d-62431aec9488 admin admin] RBAC: Authorizing `identity:create_grant(project_id=77275929094b47d291adf3fe1776f385, user_id=4cce43a9515d48f89a629be170d4da2d, role_id=ead5fac86a0747ea8823a82efcea8ed7)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:01:57.633241 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7d23529-daa3-4a50-9b1d-62431aec9488 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:01:57.647998 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 823/1636] 10.222.0.27 () {60 vars in 1483 bytes} [Tue Aug 30 02:01:57 2022] PUT /identity/v3/projects/77275929094b47d291adf3fe1776f385/users/4cce43a9515d48f89a629be170d4da2d/roles/ead5fac86a0747ea8823a82efcea8ed7 => generated 0 bytes in 42 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 02:01:57.649486 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79c49926-f5a5-4e23-9bc7-a3020d2bef4b None None] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:01:57.649896 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79c49926-f5a5-4e23-9bc7-a3020d2bef4b None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:01:57.650166 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79c49926-f5a5-4e23-9bc7-a3020d2bef4b None None] PATH_INFO: `/` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:01:57.657274 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 824/1637] 10.222.0.27 () {54 vars in 912 bytes} [Tue Aug 30 02:01:57 2022] GET /identity => generated 270 bytes in 9 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 02:01:57.659169 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79c49926-f5a5-4e23-9bc7-a3020d2bef4b None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:01:57.662184 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8afe754-d638-4151-b3ce-f8e42ab98da4 admin admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:01:57.753483 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-938a22e5-136c-4fc2-91e7-915e996aabe3 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ab936b3767a8417aa80c8c8f1b84b667', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4562f3951fbe465280073e60a11e44e2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:01:57.754525 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-938a22e5-136c-4fc2-91e7-915e996aabe3 service cinder] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:01:57.755606 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-938a22e5-136c-4fc2-91e7-915e996aabe3 service cinder] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:01:57.756160 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-938a22e5-136c-4fc2-91e7-915e996aabe3 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:01:57.764573 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79c49926-f5a5-4e23-9bc7-a3020d2bef4b None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:01:57.776192 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54bbf3d8-39cb-4476-84e1-76e35b371530 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ab936b3767a8417aa80c8c8f1b84b667', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4562f3951fbe465280073e60a11e44e2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:01:57.777618 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54bbf3d8-39cb-4476-84e1-76e35b371530 service cinder] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:01:57.778056 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54bbf3d8-39cb-4476-84e1-76e35b371530 service cinder] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:01:57.778325 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54bbf3d8-39cb-4476-84e1-76e35b371530 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:01:57.827573 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:01:57.848856 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:01:57.850896 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-938a22e5-136c-4fc2-91e7-915e996aabe3 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:01:57.858286 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54bbf3d8-39cb-4476-84e1-76e35b371530 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:01:57.858458 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-938a22e5-136c-4fc2-91e7-915e996aabe3 service cinder] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:01:57.858564 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54bbf3d8-39cb-4476-84e1-76e35b371530 service cinder] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:01:57.865106 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 825/1638] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Aug 30 02:01:57 2022] GET /identity/v3/auth/tokens => generated 3323 bytes in 207 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 02:01:57.865766 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 814/1639] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Aug 30 02:01:57 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 314 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 02:01:57.866611 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57b317a7-30ce-4bf1-932b-422ef414d62c None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:01:57.866967 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57b317a7-30ce-4bf1-932b-422ef414d62c None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:01:57.867175 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e808b54-8631-46f7-8787-ad07393fb8b7 None None] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:01:57.867354 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57b317a7-30ce-4bf1-932b-422ef414d62c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:01:57.867486 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e808b54-8631-46f7-8787-ad07393fb8b7 None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:01:57.867805 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e808b54-8631-46f7-8787-ad07393fb8b7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:01:57.897574 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.auth.core [None req-0e808b54-8631-46f7-8787-ad07393fb8b7 None None] MFA Rules not processed for user `d64f2e71361644589bec8a06e6eceed1`. Rule list: `[]` (Enabled: `True`). {{(pid=81578) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:01:57.899751 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-0e808b54-8631-46f7-8787-ad07393fb8b7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:01:57.901321 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-57b317a7-30ce-4bf1-932b-422ef414d62c None None] MFA Rules not processed for user `4cce43a9515d48f89a629be170d4da2d`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:01:57.902637 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-57b317a7-30ce-4bf1-932b-422ef414d62c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:01:57.912560 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 815/1640] 10.222.0.27 () {58 vars in 1009 bytes} [Tue Aug 30 02:01:57 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 46 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 02:01:58.016573 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 826/1641] 10.222.0.27 () {58 vars in 958 bytes} [Tue Aug 30 02:01:57 2022] POST /identity/v3/auth/tokens => generated 3331 bytes in 145 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 02:01:58.024997 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e808b54-8631-46f7-8787-ad07393fb8b7 None None] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:01:58.034467 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e808b54-8631-46f7-8787-ad07393fb8b7 None None] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:01:58.047301 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12be65bb-6285-48d6-9026-1a9505a837d8 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ab936b3767a8417aa80c8c8f1b84b667', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4562f3951fbe465280073e60a11e44e2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:01:58.048061 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12be65bb-6285-48d6-9026-1a9505a837d8 service cinder] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:01:58.048341 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12be65bb-6285-48d6-9026-1a9505a837d8 service cinder] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:01:58.048615 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12be65bb-6285-48d6-9026-1a9505a837d8 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:01:58.067969 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12be65bb-6285-48d6-9026-1a9505a837d8 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:01:58.069406 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12be65bb-6285-48d6-9026-1a9505a837d8 service cinder] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:01:58.083758 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 816/1642] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Aug 30 02:01:58 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 61 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 30 02:01:58.163924 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57b317a7-30ce-4bf1-932b-422ef414d62c None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:01:58.176961 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57b317a7-30ce-4bf1-932b-422ef414d62c None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:01:58.180829 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f91ff9c-2486-4f9a-b88f-176570896bc3 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ab936b3767a8417aa80c8c8f1b84b667', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4562f3951fbe465280073e60a11e44e2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:01:58.200256 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f91ff9c-2486-4f9a-b88f-176570896bc3 service cinder] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:01:58.200256 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f91ff9c-2486-4f9a-b88f-176570896bc3 service cinder] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:01:58.200256 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f91ff9c-2486-4f9a-b88f-176570896bc3 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:01:58.200256 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f91ff9c-2486-4f9a-b88f-176570896bc3 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:01:58.200256 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f91ff9c-2486-4f9a-b88f-176570896bc3 service cinder] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:01:58.208126 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 827/1643] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Aug 30 02:01:58 2022] GET /identity/v3/auth/tokens => generated 3331 bytes in 45 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 02:01:58.253143 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12be65bb-6285-48d6-9026-1a9505a837d8 service cinder] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:01:58.649734 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12be65bb-6285-48d6-9026-1a9505a837d8 service cinder] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:01:58.649734 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0aeb3db6-9daa-4f2c-b4ce-d27a015a8900 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:01:58.649734 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0aeb3db6-9daa-4f2c-b4ce-d27a015a8900 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:01:58.649734 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0aeb3db6-9daa-4f2c-b4ce-d27a015a8900 admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:01:58.657117 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0aeb3db6-9daa-4f2c-b4ce-d27a015a8900 admin admin] PATH_INFO: `/v3/users/26c77ddb8a184d1ea623f379d492768b` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:01:58.657117 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0aeb3db6-9daa-4f2c-b4ce-d27a015a8900 admin admin] RBAC: Authorizing `identity:delete_user(user_id=26c77ddb8a184d1ea623f379d492768b)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:01:58.657117 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0aeb3db6-9daa-4f2c-b4ce-d27a015a8900 admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:01:58.662621 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f91ff9c-2486-4f9a-b88f-176570896bc3 service cinder] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:01:58.662621 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f91ff9c-2486-4f9a-b88f-176570896bc3 service cinder] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:01:58.662621 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a18202a8-99da-4d89-ae42-b6adbffb6121 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ab936b3767a8417aa80c8c8f1b84b667', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4562f3951fbe465280073e60a11e44e2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:01:58.662621 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a18202a8-99da-4d89-ae42-b6adbffb6121 service cinder] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:01:58.667958 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a18202a8-99da-4d89-ae42-b6adbffb6121 service cinder] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:01:58.667958 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a18202a8-99da-4d89-ae42-b6adbffb6121 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:01:58.718471 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a18202a8-99da-4d89-ae42-b6adbffb6121 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:01:58.720031 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a18202a8-99da-4d89-ae42-b6adbffb6121 service cinder] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:01:58.730451 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 828/1644] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Aug 30 02:01:58 2022] GET /identity/v3/auth/tokens => generated 3256 bytes in 128 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 02:01:58.731388 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a18202a8-99da-4d89-ae42-b6adbffb6121 service cinder] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:01:58.743051 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a18202a8-99da-4d89-ae42-b6adbffb6121 service cinder] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:01:58.749640 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce147b20-9d4b-478a-b014-e5ff6abbaadc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:01:58.753762 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce147b20-9d4b-478a-b014-e5ff6abbaadc admin admin] REQUEST_METHOD: `DELETE` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:01:58.753762 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce147b20-9d4b-478a-b014-e5ff6abbaadc admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:01:58.753762 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce147b20-9d4b-478a-b014-e5ff6abbaadc admin admin] PATH_INFO: `/v3/users/8019b993d3b74f13ad20564a86f39ab3` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:01:58.753762 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce147b20-9d4b-478a-b014-e5ff6abbaadc admin admin] RBAC: Authorizing `identity:delete_user(user_id=8019b993d3b74f13ad20564a86f39ab3)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:01:58.753762 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: WARNING py.warnings [None req-ce147b20-9d4b-478a-b014-e5ff6abbaadc admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:delete_user": "(role:admin and system_scope:all) or (role:admin and token.domain.id:%(target.user.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Aug 30 02:01:58.753762 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: warnings.warn(msg) Aug 30 02:01:58.753762 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: Aug 30 02:01:58.754208 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce147b20-9d4b-478a-b014-e5ff6abbaadc admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:01:58.854484 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-0aeb3db6-9daa-4f2c-b4ce-d27a015a8900 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '26c77ddb8a184d1ea623f379d492768b'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:01:58.855939 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-0aeb3db6-9daa-4f2c-b4ce-d27a015a8900 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '26c77ddb8a184d1ea623f379d492768b'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:01:59.007095 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-0aeb3db6-9daa-4f2c-b4ce-d27a015a8900 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '26c77ddb8a184d1ea623f379d492768b'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:01:59.007888 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-0aeb3db6-9daa-4f2c-b4ce-d27a015a8900 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '26c77ddb8a184d1ea623f379d492768b'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:01:59.026096 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-0aeb3db6-9daa-4f2c-b4ce-d27a015a8900 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '26c77ddb8a184d1ea623f379d492768b'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:01:59.047113 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 817/1645] 10.222.0.27 () {58 vars in 1224 bytes} [Tue Aug 30 02:01:58 2022] DELETE /identity/v3/users/26c77ddb8a184d1ea623f379d492768b => generated 0 bytes in 795 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 02:01:59.051594 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e91bdacd-7912-4b72-8cee-14ba6e5ab0e9 None None] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:01:59.051898 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e91bdacd-7912-4b72-8cee-14ba6e5ab0e9 None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:01:59.057052 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e91bdacd-7912-4b72-8cee-14ba6e5ab0e9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:01:59.073988 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.auth.core [None req-e91bdacd-7912-4b72-8cee-14ba6e5ab0e9 None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81578) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:01:59.076169 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-e91bdacd-7912-4b72-8cee-14ba6e5ab0e9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:01:59.128223 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-ce147b20-9d4b-478a-b014-e5ff6abbaadc admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '8019b993d3b74f13ad20564a86f39ab3'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:01:59.128917 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-ce147b20-9d4b-478a-b014-e5ff6abbaadc admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '8019b993d3b74f13ad20564a86f39ab3'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:01:59.129613 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-ce147b20-9d4b-478a-b014-e5ff6abbaadc admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '8019b993d3b74f13ad20564a86f39ab3'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:01:59.152632 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-ce147b20-9d4b-478a-b014-e5ff6abbaadc admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '8019b993d3b74f13ad20564a86f39ab3'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:01:59.171988 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:01:59.172173 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-ce147b20-9d4b-478a-b014-e5ff6abbaadc admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '8019b993d3b74f13ad20564a86f39ab3'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:01:59.224310 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=92,keystone:DELETE=8 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:01:59.232047 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 818/1646] 10.222.0.27 () {58 vars in 958 bytes} [Tue Aug 30 02:01:59 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 181 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 30 02:01:59.240886 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e91bdacd-7912-4b72-8cee-14ba6e5ab0e9 None None] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:01:59.242434 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-e91bdacd-7912-4b72-8cee-14ba6e5ab0e9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:01:59.294236 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 829/1647] 10.222.0.27 () {58 vars in 1224 bytes} [Tue Aug 30 02:01:58 2022] DELETE /identity/v3/users/8019b993d3b74f13ad20564a86f39ab3 => generated 0 bytes in 563 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 02:01:59.299853 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-962b5d3c-4ed9-4057-93f5-d476794ceb5e None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:01:59.300225 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-962b5d3c-4ed9-4057-93f5-d476794ceb5e None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:01:59.301390 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-962b5d3c-4ed9-4057-93f5-d476794ceb5e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:01:59.355063 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-962b5d3c-4ed9-4057-93f5-d476794ceb5e None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:01:59.356301 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-962b5d3c-4ed9-4057-93f5-d476794ceb5e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:01:59.414602 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e91bdacd-7912-4b72-8cee-14ba6e5ab0e9 None None] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:01:59.579234 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 830/1648] 10.222.0.27 () {58 vars in 958 bytes} [Tue Aug 30 02:01:59 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 279 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 02:01:59.581635 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38dd2e28-a1e3-4d9a-a2b8-bf81f1e11855 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f3ea1580c56f4814a92bbb2acf542905', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4562f3951fbe465280073e60a11e44e2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:01:59.582277 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38dd2e28-a1e3-4d9a-a2b8-bf81f1e11855 service neutron] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:01:59.582561 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38dd2e28-a1e3-4d9a-a2b8-bf81f1e11855 service neutron] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:01:59.584897 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38dd2e28-a1e3-4d9a-a2b8-bf81f1e11855 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:01:59.597870 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-962b5d3c-4ed9-4057-93f5-d476794ceb5e None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:01:59.613403 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-962b5d3c-4ed9-4057-93f5-d476794ceb5e None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:01:59.619890 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38dd2e28-a1e3-4d9a-a2b8-bf81f1e11855 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:01:59.626080 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38dd2e28-a1e3-4d9a-a2b8-bf81f1e11855 service neutron] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:01:59.631514 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eae61f22-d005-4153-9fcc-79402f88ffa6 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f3ea1580c56f4814a92bbb2acf542905', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4562f3951fbe465280073e60a11e44e2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:01:59.633930 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eae61f22-d005-4153-9fcc-79402f88ffa6 service neutron] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:01:59.634376 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eae61f22-d005-4153-9fcc-79402f88ffa6 service neutron] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:01:59.634655 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eae61f22-d005-4153-9fcc-79402f88ffa6 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:01:59.663745 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 819/1649] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Aug 30 02:01:59 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 423 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 30 02:01:59.670589 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eae61f22-d005-4153-9fcc-79402f88ffa6 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:01:59.670589 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eae61f22-d005-4153-9fcc-79402f88ffa6 service neutron] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:01:59.692877 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 831/1650] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Aug 30 02:01:59 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 95 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 02:02:00.809778 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38dd2e28-a1e3-4d9a-a2b8-bf81f1e11855 service neutron] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:02:00.811915 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-38dd2e28-a1e3-4d9a-a2b8-bf81f1e11855 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:02:00.834427 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38dd2e28-a1e3-4d9a-a2b8-bf81f1e11855 service neutron] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:02:00.851081 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46d06b46-149d-4565-a8ca-94677114a437 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:02:00.851741 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46d06b46-149d-4565-a8ca-94677114a437 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:02:00.852250 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46d06b46-149d-4565-a8ca-94677114a437 admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:02:00.852399 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46d06b46-149d-4565-a8ca-94677114a437 admin admin] PATH_INFO: `/v3/projects/a06899b187644ff4a2c5a5a38d0ab751` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:02:00.853621 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46d06b46-149d-4565-a8ca-94677114a437 admin admin] RBAC: Authorizing `identity:delete_project(project_id=a06899b187644ff4a2c5a5a38d0ab751)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:02:00.854637 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: WARNING py.warnings [None req-46d06b46-149d-4565-a8ca-94677114a437 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:delete_project": "(role:admin and system_scope:all) or (role:admin and domain_id:%(target.project.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Aug 30 02:02:00.854637 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: warnings.warn(msg) Aug 30 02:02:00.854637 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: Aug 30 02:02:00.856240 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46d06b46-149d-4565-a8ca-94677114a437 admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:02:00.874106 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eae61f22-d005-4153-9fcc-79402f88ffa6 service neutron] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:02:00.874106 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-eae61f22-d005-4153-9fcc-79402f88ffa6 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:02:00.908614 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eae61f22-d005-4153-9fcc-79402f88ffa6 service neutron] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:02:00.931361 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9cb9983-558c-4998-a2b6-71420e9100ab admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:02:00.931882 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9cb9983-558c-4998-a2b6-71420e9100ab admin admin] REQUEST_METHOD: `DELETE` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:02:00.932165 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9cb9983-558c-4998-a2b6-71420e9100ab admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:02:00.932440 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9cb9983-558c-4998-a2b6-71420e9100ab admin admin] PATH_INFO: `/v3/projects/c28e611dd60f47dbb3a396a2b59406c7` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:02:00.934200 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9cb9983-558c-4998-a2b6-71420e9100ab admin admin] RBAC: Authorizing `identity:delete_project(project_id=c28e611dd60f47dbb3a396a2b59406c7)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:02:00.943043 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9cb9983-558c-4998-a2b6-71420e9100ab admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:02:01.074142 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-46d06b46-149d-4565-a8ca-94677114a437 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'a06899b187644ff4a2c5a5a38d0ab751', 'request_id': 'req-46d06b46-149d-4565-a8ca-94677114a437'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:02:01.101756 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-46d06b46-149d-4565-a8ca-94677114a437 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'a06899b187644ff4a2c5a5a38d0ab751', 'request_id': 'req-46d06b46-149d-4565-a8ca-94677114a437'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:02:01.136216 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-46d06b46-149d-4565-a8ca-94677114a437 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'a06899b187644ff4a2c5a5a38d0ab751', 'request_id': 'req-46d06b46-149d-4565-a8ca-94677114a437'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:02:01.191492 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-d9cb9983-558c-4998-a2b6-71420e9100ab admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'c28e611dd60f47dbb3a396a2b59406c7', 'request_id': 'req-d9cb9983-558c-4998-a2b6-71420e9100ab'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:02:01.205516 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-d9cb9983-558c-4998-a2b6-71420e9100ab admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'c28e611dd60f47dbb3a396a2b59406c7', 'request_id': 'req-d9cb9983-558c-4998-a2b6-71420e9100ab'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:02:01.237351 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-46d06b46-149d-4565-a8ca-94677114a437 admin admin] The token cache is being invalidate because project a06899b187644ff4a2c5a5a38d0ab751 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81578) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 30 02:02:01.237673 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-46d06b46-149d-4565-a8ca-94677114a437 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:02:01.237761 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-d9cb9983-558c-4998-a2b6-71420e9100ab admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'c28e611dd60f47dbb3a396a2b59406c7', 'request_id': 'req-d9cb9983-558c-4998-a2b6-71420e9100ab'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:02:01.240276 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-46d06b46-149d-4565-a8ca-94677114a437 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:02:01.242481 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 820/1651] 10.222.0.27 () {58 vars in 1233 bytes} [Tue Aug 30 02:02:00 2022] DELETE /identity/v3/projects/a06899b187644ff4a2c5a5a38d0ab751 => generated 0 bytes in 433 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 02:02:01.247202 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46d06b46-149d-4565-a8ca-94677114a437 admin admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:02:01.252745 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-46d06b46-149d-4565-a8ca-94677114a437 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:02:01.300389 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:02:01.338694 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-d9cb9983-558c-4998-a2b6-71420e9100ab admin admin] The token cache is being invalidate because project c28e611dd60f47dbb3a396a2b59406c7 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81577) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 30 02:02:01.338948 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-d9cb9983-558c-4998-a2b6-71420e9100ab admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:02:01.340762 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-d9cb9983-558c-4998-a2b6-71420e9100ab admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:02:01.345769 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 832/1652] 10.222.0.27 () {58 vars in 1233 bytes} [Tue Aug 30 02:02:00 2022] DELETE /identity/v3/projects/c28e611dd60f47dbb3a396a2b59406c7 => generated 0 bytes in 474 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 02:02:01.353157 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ae149d5-f65f-4fef-9e3b-d1483cba5f08 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:02:01.354362 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ae149d5-f65f-4fef-9e3b-d1483cba5f08 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:02:01.356606 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ae149d5-f65f-4fef-9e3b-d1483cba5f08 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:02:01.487670 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-0ae149d5-f65f-4fef-9e3b-d1483cba5f08 None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:02:01.494488 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-0ae149d5-f65f-4fef-9e3b-d1483cba5f08 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:02:01.514011 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46d06b46-149d-4565-a8ca-94677114a437 admin admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:02:01.601270 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:DELETE=9,keystone:SELECT=89,keystone:INSERT=2 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:02:01.673183 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 833/1653] 10.222.0.27 () {58 vars in 958 bytes} [Tue Aug 30 02:02:01 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 312 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 30 02:02:01.673344 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4c8c39a-b882-4291-a212-28d3801b3532 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:02:01.673344 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4c8c39a-b882-4291-a212-28d3801b3532 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:02:01.673344 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4c8c39a-b882-4291-a212-28d3801b3532 admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:02:01.673344 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4c8c39a-b882-4291-a212-28d3801b3532 admin admin] PATH_INFO: `/v3/users/4cce43a9515d48f89a629be170d4da2d` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:02:01.673648 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d45eb7d7-dd5f-43c4-bef6-5e92fd9e75df None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:02:01.673648 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d45eb7d7-dd5f-43c4-bef6-5e92fd9e75df None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:02:01.673648 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d45eb7d7-dd5f-43c4-bef6-5e92fd9e75df None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:02:01.674845 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4c8c39a-b882-4291-a212-28d3801b3532 admin admin] RBAC: Authorizing `identity:delete_user(user_id=4cce43a9515d48f89a629be170d4da2d)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:02:01.680474 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4c8c39a-b882-4291-a212-28d3801b3532 admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:02:01.736668 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-d45eb7d7-dd5f-43c4-bef6-5e92fd9e75df None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:02:01.739027 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-d45eb7d7-dd5f-43c4-bef6-5e92fd9e75df None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:02:01.759781 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 834/1654] 10.222.0.27 () {58 vars in 958 bytes} [Tue Aug 30 02:02:01 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 02:02:01.761008 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d45eb7d7-dd5f-43c4-bef6-5e92fd9e75df None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:02:01.782914 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d45eb7d7-dd5f-43c4-bef6-5e92fd9e75df None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:02:01.789107 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91b00bb2-50b2-4f64-95d2-861cf5fb5f6c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:02:01.789877 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91b00bb2-50b2-4f64-95d2-861cf5fb5f6c admin admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:02:01.790366 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91b00bb2-50b2-4f64-95d2-861cf5fb5f6c admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:02:01.790665 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91b00bb2-50b2-4f64-95d2-861cf5fb5f6c admin admin] PATH_INFO: `/v3/domains` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:02:01.791312 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91b00bb2-50b2-4f64-95d2-861cf5fb5f6c admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 02:02:01.791619 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91b00bb2-50b2-4f64-95d2-861cf5fb5f6c admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:02:01.794235 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91b00bb2-50b2-4f64-95d2-861cf5fb5f6c admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:02:01.809767 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 835/1655] 10.222.0.27 () {58 vars in 1153 bytes} [Tue Aug 30 02:02:01 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 49 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 02:02:01.814931 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91b00bb2-50b2-4f64-95d2-861cf5fb5f6c admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:02:01.825307 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91b00bb2-50b2-4f64-95d2-861cf5fb5f6c admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:02:01.831554 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e8ecfe3-cd53-42e8-abec-8157a8ddc57b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:02:01.832329 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e8ecfe3-cd53-42e8-abec-8157a8ddc57b admin admin] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:02:01.832617 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e8ecfe3-cd53-42e8-abec-8157a8ddc57b admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:02:01.833101 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e8ecfe3-cd53-42e8-abec-8157a8ddc57b admin admin] PATH_INFO: `/v3/projects` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:02:01.833867 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e8ecfe3-cd53-42e8-abec-8157a8ddc57b admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:02:01.836229 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e8ecfe3-cd53-42e8-abec-8157a8ddc57b admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:02:01.941822 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-b4c8c39a-b882-4291-a212-28d3801b3532 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '4cce43a9515d48f89a629be170d4da2d'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:02:01.942829 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-b4c8c39a-b882-4291-a212-28d3801b3532 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '4cce43a9515d48f89a629be170d4da2d'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:02:01.975348 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 836/1656] 10.222.0.27 () {60 vars in 1153 bytes} [Tue Aug 30 02:02:01 2022] POST /identity/v3/projects => generated 355 bytes in 161 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 30 02:02:01.982879 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e8ecfe3-cd53-42e8-abec-8157a8ddc57b admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:02:02.070723 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-b4c8c39a-b882-4291-a212-28d3801b3532 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '4cce43a9515d48f89a629be170d4da2d'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:02:02.071704 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-b4c8c39a-b882-4291-a212-28d3801b3532 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '4cce43a9515d48f89a629be170d4da2d'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:02:02.084224 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-b4c8c39a-b882-4291-a212-28d3801b3532 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '4cce43a9515d48f89a629be170d4da2d'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:02:02.178651 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 821/1657] 10.222.0.27 () {58 vars in 1224 bytes} [Tue Aug 30 02:02:01 2022] DELETE /identity/v3/users/4cce43a9515d48f89a629be170d4da2d => generated 0 bytes in 915 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 02:02:02.226620 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e8ecfe3-cd53-42e8-abec-8157a8ddc57b admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:02:02.431377 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ba85900-9644-44c1-829a-6f31865b6023 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:02:02.432387 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ba85900-9644-44c1-829a-6f31865b6023 admin admin] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:02:02.432674 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ba85900-9644-44c1-829a-6f31865b6023 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:02:02.432952 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ba85900-9644-44c1-829a-6f31865b6023 admin admin] PATH_INFO: `/v3/users` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:02:02.433702 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ba85900-9644-44c1-829a-6f31865b6023 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:02:02.436652 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ba85900-9644-44c1-829a-6f31865b6023 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:02:02.581888 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 837/1658] 10.222.0.27 () {60 vars in 1144 bytes} [Tue Aug 30 02:02:01 2022] POST /identity/v3/users => generated 278 bytes in 603 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 30 02:02:02.585501 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4c8c39a-b882-4291-a212-28d3801b3532 admin admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:02:02.587054 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-b4c8c39a-b882-4291-a212-28d3801b3532 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:02:02.610595 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4c8c39a-b882-4291-a212-28d3801b3532 admin admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:02:02.627003 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93ab2350-e2f8-4b4a-ae81-c6a468ac0412 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:02:02.627992 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93ab2350-e2f8-4b4a-ae81-c6a468ac0412 admin admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:02:02.628587 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93ab2350-e2f8-4b4a-ae81-c6a468ac0412 admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:02:02.629023 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93ab2350-e2f8-4b4a-ae81-c6a468ac0412 admin admin] PATH_INFO: `/v3/roles` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:02:02.629914 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93ab2350-e2f8-4b4a-ae81-c6a468ac0412 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:02:02.633577 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93ab2350-e2f8-4b4a-ae81-c6a468ac0412 admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:02:02.649190 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 822/1659] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Aug 30 02:02:02 2022] GET /identity/v3/roles => generated 1420 bytes in 64 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 02:02:02.653173 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ba85900-9644-44c1-829a-6f31865b6023 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:02:02.664906 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93ab2350-e2f8-4b4a-ae81-c6a468ac0412 admin admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:02:02.671102 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-93ab2350-e2f8-4b4a-ae81-c6a468ac0412 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:02:02.674090 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ba85900-9644-44c1-829a-6f31865b6023 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:02:02.699764 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f426d1d-a907-4de4-a645-f3583408d088 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:02:02.700506 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f426d1d-a907-4de4-a645-f3583408d088 admin admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:02:02.700791 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f426d1d-a907-4de4-a645-f3583408d088 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:02:02.704931 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f426d1d-a907-4de4-a645-f3583408d088 admin admin] PATH_INFO: `/v3/roles` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:02:02.706485 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f426d1d-a907-4de4-a645-f3583408d088 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:02:02.714311 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f426d1d-a907-4de4-a645-f3583408d088 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:02:02.727015 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93ab2350-e2f8-4b4a-ae81-c6a468ac0412 admin admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:02:02.751040 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-976b2ee6-6492-4d69-8543-4926b7e904dc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:02:02.751688 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-976b2ee6-6492-4d69-8543-4926b7e904dc admin admin] REQUEST_METHOD: `DELETE` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:02:02.754462 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-976b2ee6-6492-4d69-8543-4926b7e904dc admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:02:02.754891 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-976b2ee6-6492-4d69-8543-4926b7e904dc admin admin] PATH_INFO: `/v3/users/9bb1df3fe2214e9da8e72b1047d569f2` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:02:02.768290 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-976b2ee6-6492-4d69-8543-4926b7e904dc admin admin] RBAC: Authorizing `identity:delete_user(user_id=9bb1df3fe2214e9da8e72b1047d569f2)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:02:02.770431 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-976b2ee6-6492-4d69-8543-4926b7e904dc admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:02:02.788433 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 838/1660] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Aug 30 02:02:02 2022] GET /identity/v3/roles => generated 1420 bytes in 132 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 02:02:02.788433 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f426d1d-a907-4de4-a645-f3583408d088 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:02:02.804268 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f426d1d-a907-4de4-a645-f3583408d088 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:02:02.813779 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1cb1233-7224-4aa8-9b1c-0a7537e27d55 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:02:02.815461 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1cb1233-7224-4aa8-9b1c-0a7537e27d55 admin admin] REQUEST_METHOD: `PUT` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:02:02.815800 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1cb1233-7224-4aa8-9b1c-0a7537e27d55 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:02:02.816172 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1cb1233-7224-4aa8-9b1c-0a7537e27d55 admin admin] PATH_INFO: `/v3/projects/3ce8ab0cd06941d79f67cc35a06bb2e6/users/a1614048b7d0464a9cdf7dec2c6be828/roles/1fb81afcb9f245f3b989add94598a063` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:02:02.854115 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1cb1233-7224-4aa8-9b1c-0a7537e27d55 admin admin] RBAC: Authorizing `identity:create_grant(project_id=3ce8ab0cd06941d79f67cc35a06bb2e6, user_id=a1614048b7d0464a9cdf7dec2c6be828, role_id=1fb81afcb9f245f3b989add94598a063)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:02:02.857054 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1cb1233-7224-4aa8-9b1c-0a7537e27d55 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:02:02.993309 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 839/1661] 10.222.0.27 () {60 vars in 1483 bytes} [Tue Aug 30 02:02:02 2022] PUT /identity/v3/projects/3ce8ab0cd06941d79f67cc35a06bb2e6/users/a1614048b7d0464a9cdf7dec2c6be828/roles/1fb81afcb9f245f3b989add94598a063 => generated 0 bytes in 207 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 02:02:03.013494 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1704edc8-23a3-4160-8b02-bdfee8d4ccd8 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:02:03.013807 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1704edc8-23a3-4160-8b02-bdfee8d4ccd8 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:02:03.014141 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1704edc8-23a3-4160-8b02-bdfee8d4ccd8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:02:03.081198 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-1704edc8-23a3-4160-8b02-bdfee8d4ccd8 None None] MFA Rules not processed for user `a1614048b7d0464a9cdf7dec2c6be828`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:02:03.083913 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-1704edc8-23a3-4160-8b02-bdfee8d4ccd8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:02:03.104553 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-976b2ee6-6492-4d69-8543-4926b7e904dc admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '9bb1df3fe2214e9da8e72b1047d569f2'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:02:03.106768 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-976b2ee6-6492-4d69-8543-4926b7e904dc admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '9bb1df3fe2214e9da8e72b1047d569f2'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:02:03.158470 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=84,keystone:DELETE=15,keystone:INSERT=1 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:02:03.178300 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-976b2ee6-6492-4d69-8543-4926b7e904dc admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '9bb1df3fe2214e9da8e72b1047d569f2'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:02:03.179092 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-976b2ee6-6492-4d69-8543-4926b7e904dc admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '9bb1df3fe2214e9da8e72b1047d569f2'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:02:03.191345 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-976b2ee6-6492-4d69-8543-4926b7e904dc admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '9bb1df3fe2214e9da8e72b1047d569f2'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:02:03.230987 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 823/1662] 10.222.0.27 () {58 vars in 1224 bytes} [Tue Aug 30 02:02:02 2022] DELETE /identity/v3/users/9bb1df3fe2214e9da8e72b1047d569f2 => generated 0 bytes in 562 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 02:02:03.232775 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cbb62dd-9d9c-4f29-b632-e996fb68574c None None] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:02:03.233288 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cbb62dd-9d9c-4f29-b632-e996fb68574c None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:02:03.233599 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cbb62dd-9d9c-4f29-b632-e996fb68574c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:02:03.265643 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.auth.core [None req-0cbb62dd-9d9c-4f29-b632-e996fb68574c None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81578) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:02:03.265931 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 840/1663] 10.222.0.27 () {58 vars in 958 bytes} [Tue Aug 30 02:02:03 2022] POST /identity/v3/auth/tokens => generated 3232 bytes in 255 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 02:02:03.267142 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-0cbb62dd-9d9c-4f29-b632-e996fb68574c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:02:03.270478 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1704edc8-23a3-4160-8b02-bdfee8d4ccd8 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:02:03.275605 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-1704edc8-23a3-4160-8b02-bdfee8d4ccd8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:02:03.413320 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:02:03.426411 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1704edc8-23a3-4160-8b02-bdfee8d4ccd8 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:02:03.543800 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 824/1664] 10.222.0.27 () {58 vars in 958 bytes} [Tue Aug 30 02:02:03 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 311 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 02:02:03.544897 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cbb62dd-9d9c-4f29-b632-e996fb68574c None None] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:02:03.556774 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-0cbb62dd-9d9c-4f29-b632-e996fb68574c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:02:03.615446 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cbb62dd-9d9c-4f29-b632-e996fb68574c None None] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:02:03.634145 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37ca9ae9-4dd1-47f9-98d3-8390e3690823 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:02:03.634809 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37ca9ae9-4dd1-47f9-98d3-8390e3690823 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:02:03.635090 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37ca9ae9-4dd1-47f9-98d3-8390e3690823 admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:02:03.635359 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37ca9ae9-4dd1-47f9-98d3-8390e3690823 admin admin] PATH_INFO: `/v3/projects/77275929094b47d291adf3fe1776f385` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:02:03.642385 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37ca9ae9-4dd1-47f9-98d3-8390e3690823 admin admin] RBAC: Authorizing `identity:delete_project(project_id=77275929094b47d291adf3fe1776f385)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:02:03.644331 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37ca9ae9-4dd1-47f9-98d3-8390e3690823 admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:02:03.651838 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e7b0846-20b3-4013-a6f0-95cb470ada3e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:02:03.653874 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e7b0846-20b3-4013-a6f0-95cb470ada3e admin admin] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:02:03.653874 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e7b0846-20b3-4013-a6f0-95cb470ada3e admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:02:03.653874 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e7b0846-20b3-4013-a6f0-95cb470ada3e admin admin] PATH_INFO: `/v3/projects` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:02:03.653874 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e7b0846-20b3-4013-a6f0-95cb470ada3e admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:02:03.655826 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e7b0846-20b3-4013-a6f0-95cb470ada3e admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:02:03.806814 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 841/1665] 10.222.0.27 () {60 vars in 1153 bytes} [Tue Aug 30 02:02:03 2022] POST /identity/v3/projects => generated 357 bytes in 534 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 02:02:03.806814 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e7b0846-20b3-4013-a6f0-95cb470ada3e admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:02:03.815535 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-7e7b0846-20b3-4013-a6f0-95cb470ada3e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:02:03.868585 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-37ca9ae9-4dd1-47f9-98d3-8390e3690823 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '77275929094b47d291adf3fe1776f385', 'request_id': 'req-37ca9ae9-4dd1-47f9-98d3-8390e3690823'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:02:03.896662 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-37ca9ae9-4dd1-47f9-98d3-8390e3690823 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '77275929094b47d291adf3fe1776f385', 'request_id': 'req-37ca9ae9-4dd1-47f9-98d3-8390e3690823'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:02:03.913450 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-37ca9ae9-4dd1-47f9-98d3-8390e3690823 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '77275929094b47d291adf3fe1776f385', 'request_id': 'req-37ca9ae9-4dd1-47f9-98d3-8390e3690823'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:02:03.993178 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-37ca9ae9-4dd1-47f9-98d3-8390e3690823 admin admin] The token cache is being invalidate because project 77275929094b47d291adf3fe1776f385 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81578) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 30 02:02:03.993490 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-37ca9ae9-4dd1-47f9-98d3-8390e3690823 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:02:03.994318 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-37ca9ae9-4dd1-47f9-98d3-8390e3690823 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:02:03.997107 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 825/1666] 10.222.0.27 () {58 vars in 1233 bytes} [Tue Aug 30 02:02:03 2022] DELETE /identity/v3/projects/77275929094b47d291adf3fe1776f385 => generated 0 bytes in 453 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 02:02:03.999244 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37ca9ae9-4dd1-47f9-98d3-8390e3690823 admin admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:02:04.005834 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-37ca9ae9-4dd1-47f9-98d3-8390e3690823 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:02:04.024913 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e7b0846-20b3-4013-a6f0-95cb470ada3e admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:02:04.135683 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-446f48ce-9fc1-4f1b-872e-206bf328c081 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f3ea1580c56f4814a92bbb2acf542905', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4562f3951fbe465280073e60a11e44e2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:02:04.136403 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-446f48ce-9fc1-4f1b-872e-206bf328c081 service neutron] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:02:04.143550 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-446f48ce-9fc1-4f1b-872e-206bf328c081 service neutron] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:02:04.143550 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-446f48ce-9fc1-4f1b-872e-206bf328c081 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:02:04.145048 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-446f48ce-9fc1-4f1b-872e-206bf328c081 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:02:04.214184 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37ca9ae9-4dd1-47f9-98d3-8390e3690823 admin admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:02:04.229348 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e7d5cd6-9430-4166-a0e2-53f5d8c67af3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:02:04.231637 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e7d5cd6-9430-4166-a0e2-53f5d8c67af3 admin admin] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:02:04.231926 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e7d5cd6-9430-4166-a0e2-53f5d8c67af3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:02:04.232244 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e7d5cd6-9430-4166-a0e2-53f5d8c67af3 admin admin] PATH_INFO: `/v3/users` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:02:04.233253 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e7d5cd6-9430-4166-a0e2-53f5d8c67af3 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:02:04.243718 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e7d5cd6-9430-4166-a0e2-53f5d8c67af3 admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:02:04.302910 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-446f48ce-9fc1-4f1b-872e-206bf328c081 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:02:04.304778 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-446f48ce-9fc1-4f1b-872e-206bf328c081 service neutron] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:02:04.318158 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 842/1667] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Aug 30 02:02:03 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 513 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 02:02:04.319881 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-303df11f-2a6b-4dcd-8aef-e78221b2679f None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:02:04.322853 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-303df11f-2a6b-4dcd-8aef-e78221b2679f None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:02:04.322853 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-303df11f-2a6b-4dcd-8aef-e78221b2679f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:02:04.352285 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 826/1668] 10.222.0.27 () {60 vars in 1144 bytes} [Tue Aug 30 02:02:03 2022] POST /identity/v3/users => generated 285 bytes in 354 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 02:02:04.358207 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e7d5cd6-9430-4166-a0e2-53f5d8c67af3 admin admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:02:04.360456 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-303df11f-2a6b-4dcd-8aef-e78221b2679f None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:02:04.362205 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-303df11f-2a6b-4dcd-8aef-e78221b2679f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:02:04.383114 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 843/1669] 10.222.0.27 () {58 vars in 958 bytes} [Tue Aug 30 02:02:04 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 64 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 02:02:04.386145 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e7d5cd6-9430-4166-a0e2-53f5d8c67af3 admin admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:02:04.394776 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe34a033-8efe-420a-a000-6da8a0462c60 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:02:04.395443 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe34a033-8efe-420a-a000-6da8a0462c60 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:02:04.396025 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe34a033-8efe-420a-a000-6da8a0462c60 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:02:04.399066 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1aecf1f-5eed-41be-b732-3bc37191b7f9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:02:04.399769 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1aecf1f-5eed-41be-b732-3bc37191b7f9 admin admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:02:04.400056 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1aecf1f-5eed-41be-b732-3bc37191b7f9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:02:04.400334 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1aecf1f-5eed-41be-b732-3bc37191b7f9 admin admin] PATH_INFO: `/v3/roles` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:02:04.400989 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1aecf1f-5eed-41be-b732-3bc37191b7f9 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:02:04.411112 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1aecf1f-5eed-41be-b732-3bc37191b7f9 admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:02:04.445971 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 827/1670] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Aug 30 02:02:04 2022] GET /identity/v3/roles => generated 1420 bytes in 89 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 02:02:04.451137 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1aecf1f-5eed-41be-b732-3bc37191b7f9 admin admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:02:04.457008 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-fe34a033-8efe-420a-a000-6da8a0462c60 None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:02:04.459087 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-fe34a033-8efe-420a-a000-6da8a0462c60 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:02:04.471715 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1aecf1f-5eed-41be-b732-3bc37191b7f9 admin admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:02:04.477397 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-310d90a3-01b4-4234-9b51-f0fa029ee0af admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:02:04.478709 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-310d90a3-01b4-4234-9b51-f0fa029ee0af admin admin] REQUEST_METHOD: `PUT` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:02:04.478995 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-310d90a3-01b4-4234-9b51-f0fa029ee0af admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:02:04.479277 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-310d90a3-01b4-4234-9b51-f0fa029ee0af admin admin] PATH_INFO: `/v3/projects/a0d103ed4e9246e29c86e461cadbdee6/users/93543dd3fdad44a3b270ee7f7cdc0f86/roles/ead5fac86a0747ea8823a82efcea8ed7` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:02:04.480200 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 844/1671] 10.222.0.27 () {58 vars in 958 bytes} [Tue Aug 30 02:02:04 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 30 02:02:04.485485 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe34a033-8efe-420a-a000-6da8a0462c60 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:02:04.513085 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe34a033-8efe-420a-a000-6da8a0462c60 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:02:04.523042 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d930f51c-8b55-422f-a278-4f93e1fa8fae admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:02:04.523657 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d930f51c-8b55-422f-a278-4f93e1fa8fae admin admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:02:04.523937 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d930f51c-8b55-422f-a278-4f93e1fa8fae admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:02:04.524209 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d930f51c-8b55-422f-a278-4f93e1fa8fae admin admin] PATH_INFO: `/v3/domains` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:02:04.524784 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-310d90a3-01b4-4234-9b51-f0fa029ee0af admin admin] RBAC: Authorizing `identity:create_grant(project_id=a0d103ed4e9246e29c86e461cadbdee6, user_id=93543dd3fdad44a3b270ee7f7cdc0f86, role_id=ead5fac86a0747ea8823a82efcea8ed7)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:02:04.528879 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d930f51c-8b55-422f-a278-4f93e1fa8fae admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 02:02:04.529237 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d930f51c-8b55-422f-a278-4f93e1fa8fae admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:02:04.530495 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-310d90a3-01b4-4234-9b51-f0fa029ee0af admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:02:04.534242 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d930f51c-8b55-422f-a278-4f93e1fa8fae admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:02:04.554577 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 845/1672] 10.222.0.27 () {58 vars in 1153 bytes} [Tue Aug 30 02:02:04 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 70 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 30 02:02:04.560018 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d930f51c-8b55-422f-a278-4f93e1fa8fae admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:02:04.577261 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d930f51c-8b55-422f-a278-4f93e1fa8fae admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:02:04.598479 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6981b89a-2ff6-4d98-a13f-e9033f416153 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:02:04.599271 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6981b89a-2ff6-4d98-a13f-e9033f416153 admin admin] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:02:04.600067 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6981b89a-2ff6-4d98-a13f-e9033f416153 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:02:04.600067 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6981b89a-2ff6-4d98-a13f-e9033f416153 admin admin] PATH_INFO: `/v3/projects` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:02:04.601053 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6981b89a-2ff6-4d98-a13f-e9033f416153 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:02:04.603012 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6981b89a-2ff6-4d98-a13f-e9033f416153 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:02:04.647983 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:02:04.655674 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 828/1673] 10.222.0.27 () {60 vars in 1483 bytes} [Tue Aug 30 02:02:04 2022] PUT /identity/v3/projects/a0d103ed4e9246e29c86e461cadbdee6/users/93543dd3fdad44a3b270ee7f7cdc0f86/roles/ead5fac86a0747ea8823a82efcea8ed7 => generated 0 bytes in 209 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 02:02:04.676341 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-310d90a3-01b4-4234-9b51-f0fa029ee0af admin admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:02:04.784110 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 846/1674] 10.222.0.27 () {60 vars in 1153 bytes} [Tue Aug 30 02:02:04 2022] POST /identity/v3/projects => generated 381 bytes in 224 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 02:02:04.787788 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6981b89a-2ff6-4d98-a13f-e9033f416153 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:02:04.824661 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-310d90a3-01b4-4234-9b51-f0fa029ee0af admin admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:02:04.927305 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8d74b80-9cac-4091-87f7-e75a739d04f5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:02:04.928077 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8d74b80-9cac-4091-87f7-e75a739d04f5 admin admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:02:04.928457 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8d74b80-9cac-4091-87f7-e75a739d04f5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:02:04.928827 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8d74b80-9cac-4091-87f7-e75a739d04f5 admin admin] PATH_INFO: `/v3/roles` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:02:04.929604 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8d74b80-9cac-4091-87f7-e75a739d04f5 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:02:04.932376 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8d74b80-9cac-4091-87f7-e75a739d04f5 admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:02:04.952424 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6981b89a-2ff6-4d98-a13f-e9033f416153 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:02:04.962015 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 829/1675] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Aug 30 02:02:04 2022] GET /identity/v3/roles => generated 1420 bytes in 283 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 02:02:04.962015 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8d74b80-9cac-4091-87f7-e75a739d04f5 admin admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:02:04.962407 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb79ace3-a87c-4a36-9b7b-c5a3693f5601 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:02:04.963042 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb79ace3-a87c-4a36-9b7b-c5a3693f5601 admin admin] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:02:04.963402 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb79ace3-a87c-4a36-9b7b-c5a3693f5601 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:02:04.963745 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb79ace3-a87c-4a36-9b7b-c5a3693f5601 admin admin] PATH_INFO: `/v3/users` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:02:04.964387 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb79ace3-a87c-4a36-9b7b-c5a3693f5601 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:02:04.968220 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb79ace3-a87c-4a36-9b7b-c5a3693f5601 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:02:04.980887 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8d74b80-9cac-4091-87f7-e75a739d04f5 admin admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:02:04.993570 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23c335cc-d091-45da-8402-1825e0c528e7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:02:04.994509 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23c335cc-d091-45da-8402-1825e0c528e7 admin admin] REQUEST_METHOD: `PUT` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:02:04.994937 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23c335cc-d091-45da-8402-1825e0c528e7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:02:04.995378 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23c335cc-d091-45da-8402-1825e0c528e7 admin admin] PATH_INFO: `/v3/projects/a0d103ed4e9246e29c86e461cadbdee6/users/93543dd3fdad44a3b270ee7f7cdc0f86/roles/ead5fac86a0747ea8823a82efcea8ed7` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:02:05.003053 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23c335cc-d091-45da-8402-1825e0c528e7 admin admin] RBAC: Authorizing `identity:create_grant(project_id=a0d103ed4e9246e29c86e461cadbdee6, user_id=93543dd3fdad44a3b270ee7f7cdc0f86, role_id=ead5fac86a0747ea8823a82efcea8ed7)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:02:05.005804 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23c335cc-d091-45da-8402-1825e0c528e7 admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:02:05.039041 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 830/1676] 10.222.0.27 () {60 vars in 1483 bytes} [Tue Aug 30 02:02:04 2022] PUT /identity/v3/projects/a0d103ed4e9246e29c86e461cadbdee6/users/93543dd3fdad44a3b270ee7f7cdc0f86/roles/ead5fac86a0747ea8823a82efcea8ed7 => generated 0 bytes in 79 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 02:02:05.055982 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be960a14-d528-4c68-a61b-b67fcbb3e73b None None] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:02:05.056453 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be960a14-d528-4c68-a61b-b67fcbb3e73b None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:02:05.074105 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=86,keystone:DELETE=12,keystone:INSERT=2 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:02:05.074768 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be960a14-d528-4c68-a61b-b67fcbb3e73b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:02:05.182793 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 847/1677] 10.222.0.27 () {60 vars in 1144 bytes} [Tue Aug 30 02:02:04 2022] POST /identity/v3/users => generated 291 bytes in 395 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 02:02:05.187897 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb79ace3-a87c-4a36-9b7b-c5a3693f5601 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:02:05.195091 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.auth.core [None req-be960a14-d528-4c68-a61b-b67fcbb3e73b None None] MFA Rules not processed for user `93543dd3fdad44a3b270ee7f7cdc0f86`. Rule list: `[]` (Enabled: `True`). {{(pid=81578) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:02:05.198893 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-be960a14-d528-4c68-a61b-b67fcbb3e73b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:02:05.203168 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb79ace3-a87c-4a36-9b7b-c5a3693f5601 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:02:05.389775 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50471896-f8a1-4667-bf14-3b8f3eae14c9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:02:05.391092 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50471896-f8a1-4667-bf14-3b8f3eae14c9 admin admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:02:05.391324 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50471896-f8a1-4667-bf14-3b8f3eae14c9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:02:05.391649 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50471896-f8a1-4667-bf14-3b8f3eae14c9 admin admin] PATH_INFO: `/v3/roles` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:02:05.392371 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50471896-f8a1-4667-bf14-3b8f3eae14c9 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:02:05.398930 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50471896-f8a1-4667-bf14-3b8f3eae14c9 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:02:05.406284 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 831/1678] 10.222.0.27 () {58 vars in 958 bytes} [Tue Aug 30 02:02:05 2022] POST /identity/v3/auth/tokens => generated 3301 bytes in 358 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 02:02:05.418792 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be960a14-d528-4c68-a61b-b67fcbb3e73b None None] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:02:05.420274 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-be960a14-d528-4c68-a61b-b67fcbb3e73b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:02:05.437854 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 848/1679] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Aug 30 02:02:05 2022] GET /identity/v3/roles => generated 1420 bytes in 251 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 02:02:05.438994 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50471896-f8a1-4667-bf14-3b8f3eae14c9 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:02:05.465194 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50471896-f8a1-4667-bf14-3b8f3eae14c9 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:02:05.480983 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4c334be-ea93-49e9-9881-bf5a9326d364 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:02:05.481787 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4c334be-ea93-49e9-9881-bf5a9326d364 admin admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:02:05.482149 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4c334be-ea93-49e9-9881-bf5a9326d364 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:02:05.483198 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4c334be-ea93-49e9-9881-bf5a9326d364 admin admin] PATH_INFO: `/v3/roles` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:02:05.487809 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4c334be-ea93-49e9-9881-bf5a9326d364 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:02:05.489697 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4c334be-ea93-49e9-9881-bf5a9326d364 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:02:05.512837 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 849/1680] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Aug 30 02:02:05 2022] GET /identity/v3/roles => generated 1420 bytes in 74 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 02:02:05.520446 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4c334be-ea93-49e9-9881-bf5a9326d364 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:02:05.550011 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4c334be-ea93-49e9-9881-bf5a9326d364 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:02:05.555658 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87e6ee48-3ad0-4954-91a5-84bbc707336f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:02:05.556653 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87e6ee48-3ad0-4954-91a5-84bbc707336f admin admin] REQUEST_METHOD: `PUT` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:02:05.557159 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87e6ee48-3ad0-4954-91a5-84bbc707336f admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:02:05.557927 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87e6ee48-3ad0-4954-91a5-84bbc707336f admin admin] PATH_INFO: `/v3/projects/1c2a81204a1541909064de42ae1e95a5/users/43bdd0c830fd448db3df8b88df1281f2/roles/1fb81afcb9f245f3b989add94598a063` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:02:05.577897 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be960a14-d528-4c68-a61b-b67fcbb3e73b None None] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:02:05.585177 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51e71314-1baa-4733-b565-f285bd28633a service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ab936b3767a8417aa80c8c8f1b84b667', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4562f3951fbe465280073e60a11e44e2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:02:05.585848 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51e71314-1baa-4733-b565-f285bd28633a service cinder] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:02:05.586142 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51e71314-1baa-4733-b565-f285bd28633a service cinder] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:02:05.586431 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51e71314-1baa-4733-b565-f285bd28633a service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:02:05.590588 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87e6ee48-3ad0-4954-91a5-84bbc707336f admin admin] RBAC: Authorizing `identity:create_grant(project_id=1c2a81204a1541909064de42ae1e95a5, user_id=43bdd0c830fd448db3df8b88df1281f2, role_id=1fb81afcb9f245f3b989add94598a063)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:02:05.593932 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87e6ee48-3ad0-4954-91a5-84bbc707336f admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:02:05.615015 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51e71314-1baa-4733-b565-f285bd28633a service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:02:05.616770 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51e71314-1baa-4733-b565-f285bd28633a service cinder] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:02:05.631132 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 832/1681] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Aug 30 02:02:05 2022] GET /identity/v3/auth/tokens => generated 3301 bytes in 213 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 02:02:05.758345 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 850/1682] 10.222.0.27 () {60 vars in 1483 bytes} [Tue Aug 30 02:02:05 2022] PUT /identity/v3/projects/1c2a81204a1541909064de42ae1e95a5/users/43bdd0c830fd448db3df8b88df1281f2/roles/1fb81afcb9f245f3b989add94598a063 => generated 0 bytes in 238 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 02:02:05.770639 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70129091-bde4-4915-8b11-42d896f5daaa None None] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:02:05.770995 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70129091-bde4-4915-8b11-42d896f5daaa None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:02:05.771337 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70129091-bde4-4915-8b11-42d896f5daaa None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:02:05.815774 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87e6ee48-3ad0-4954-91a5-84bbc707336f admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:02:05.819382 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-87e6ee48-3ad0-4954-91a5-84bbc707336f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:02:05.870563 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.auth.core [None req-70129091-bde4-4915-8b11-42d896f5daaa None None] MFA Rules not processed for user `43bdd0c830fd448db3df8b88df1281f2`. Rule list: `[]` (Enabled: `True`). {{(pid=81578) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:02:05.876176 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-70129091-bde4-4915-8b11-42d896f5daaa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:02:05.907920 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87e6ee48-3ad0-4954-91a5-84bbc707336f admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:02:05.914155 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad909e3f-a616-4670-a33e-90b2b9dc780c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:02:05.914370 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad909e3f-a616-4670-a33e-90b2b9dc780c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:02:05.914532 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad909e3f-a616-4670-a33e-90b2b9dc780c admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:02:05.914806 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad909e3f-a616-4670-a33e-90b2b9dc780c admin admin] PATH_INFO: `/v3/projects/bacfc2716dad4ad991b56e781f2d05ef` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:02:05.916203 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad909e3f-a616-4670-a33e-90b2b9dc780c admin admin] RBAC: Authorizing `identity:delete_project(project_id=bacfc2716dad4ad991b56e781f2d05ef)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:02:05.918225 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad909e3f-a616-4670-a33e-90b2b9dc780c admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:02:06.008244 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:02:06.021366 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 833/1683] 10.222.0.27 () {58 vars in 958 bytes} [Tue Aug 30 02:02:05 2022] POST /identity/v3/auth/tokens => generated 3258 bytes in 253 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 02:02:06.022716 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70129091-bde4-4915-8b11-42d896f5daaa None None] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:02:06.024825 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-70129091-bde4-4915-8b11-42d896f5daaa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:02:06.083329 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-ad909e3f-a616-4670-a33e-90b2b9dc780c admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'bacfc2716dad4ad991b56e781f2d05ef', 'request_id': 'req-ad909e3f-a616-4670-a33e-90b2b9dc780c'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:02:06.096725 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-ad909e3f-a616-4670-a33e-90b2b9dc780c admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'bacfc2716dad4ad991b56e781f2d05ef', 'request_id': 'req-ad909e3f-a616-4670-a33e-90b2b9dc780c'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:02:06.110900 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-ad909e3f-a616-4670-a33e-90b2b9dc780c admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'bacfc2716dad4ad991b56e781f2d05ef', 'request_id': 'req-ad909e3f-a616-4670-a33e-90b2b9dc780c'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:02:06.117446 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70129091-bde4-4915-8b11-42d896f5daaa None None] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:02:06.193073 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-481c60e0-b4a0-4282-aab5-d58ccb7c07e3 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ab936b3767a8417aa80c8c8f1b84b667', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4562f3951fbe465280073e60a11e44e2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:02:06.194337 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-481c60e0-b4a0-4282-aab5-d58ccb7c07e3 service cinder] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:02:06.194719 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-481c60e0-b4a0-4282-aab5-d58ccb7c07e3 service cinder] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:02:06.196595 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-481c60e0-b4a0-4282-aab5-d58ccb7c07e3 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:02:06.198572 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-481c60e0-b4a0-4282-aab5-d58ccb7c07e3 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:02:06.246668 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-ad909e3f-a616-4670-a33e-90b2b9dc780c admin admin] The token cache is being invalidate because project bacfc2716dad4ad991b56e781f2d05ef was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81577) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 30 02:02:06.247123 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-ad909e3f-a616-4670-a33e-90b2b9dc780c admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:02:06.249541 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-ad909e3f-a616-4670-a33e-90b2b9dc780c admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:02:06.250697 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 851/1684] 10.222.0.27 () {58 vars in 1233 bytes} [Tue Aug 30 02:02:05 2022] DELETE /identity/v3/projects/bacfc2716dad4ad991b56e781f2d05ef => generated 0 bytes in 436 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 30 02:02:06.251861 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad909e3f-a616-4670-a33e-90b2b9dc780c admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:02:06.262706 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-ad909e3f-a616-4670-a33e-90b2b9dc780c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:02:06.349008 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-481c60e0-b4a0-4282-aab5-d58ccb7c07e3 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:02:06.352619 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-481c60e0-b4a0-4282-aab5-d58ccb7c07e3 service cinder] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:02:06.369301 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 834/1685] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Aug 30 02:02:06 2022] GET /identity/v3/auth/tokens => generated 3232 bytes in 347 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 02:02:06.375065 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85f8b355-ae24-4364-a118-be3aa8212121 None None] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:02:06.375493 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85f8b355-ae24-4364-a118-be3aa8212121 None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:02:06.375868 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85f8b355-ae24-4364-a118-be3aa8212121 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:02:06.416037 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.auth.core [None req-85f8b355-ae24-4364-a118-be3aa8212121 None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81578) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:02:06.417376 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-85f8b355-ae24-4364-a118-be3aa8212121 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:02:06.437189 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 835/1686] 10.222.0.27 () {58 vars in 958 bytes} [Tue Aug 30 02:02:06 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 67 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 02:02:06.443157 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad909e3f-a616-4670-a33e-90b2b9dc780c admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:02:06.445053 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-101820dd-0e10-46e3-9797-8b3b1775a2d2 None None] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:02:06.445590 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-101820dd-0e10-46e3-9797-8b3b1775a2d2 None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:02:06.446064 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-101820dd-0e10-46e3-9797-8b3b1775a2d2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:02:06.471201 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.auth.core [None req-101820dd-0e10-46e3-9797-8b3b1775a2d2 None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81578) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:02:06.472761 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-101820dd-0e10-46e3-9797-8b3b1775a2d2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:02:06.475883 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdb408e5-9c01-4abd-b73a-dbb72e5a1045 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:02:06.476530 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdb408e5-9c01-4abd-b73a-dbb72e5a1045 admin admin] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:02:06.476840 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdb408e5-9c01-4abd-b73a-dbb72e5a1045 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:02:06.477069 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdb408e5-9c01-4abd-b73a-dbb72e5a1045 admin admin] PATH_INFO: `/v3/projects` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:02:06.480667 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cdb408e5-9c01-4abd-b73a-dbb72e5a1045 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:02:06.484173 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cdb408e5-9c01-4abd-b73a-dbb72e5a1045 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:02:06.515056 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 836/1687] 10.222.0.27 () {58 vars in 958 bytes} [Tue Aug 30 02:02:06 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 41 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 30 02:02:06.515056 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-101820dd-0e10-46e3-9797-8b3b1775a2d2 None None] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:02:06.515056 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-101820dd-0e10-46e3-9797-8b3b1775a2d2 None None] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:02:06.515056 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ff8043f-f06a-41f8-b7cc-252da271daa4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:02:06.515550 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ff8043f-f06a-41f8-b7cc-252da271daa4 admin admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:02:06.515550 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ff8043f-f06a-41f8-b7cc-252da271daa4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:02:06.515550 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ff8043f-f06a-41f8-b7cc-252da271daa4 admin admin] PATH_INFO: `/v3/domains` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:02:06.518704 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ff8043f-f06a-41f8-b7cc-252da271daa4 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81578) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 02:02:06.521055 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ff8043f-f06a-41f8-b7cc-252da271daa4 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:02:06.523043 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ff8043f-f06a-41f8-b7cc-252da271daa4 admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:02:06.548834 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 837/1688] 10.222.0.27 () {58 vars in 1153 bytes} [Tue Aug 30 02:02:06 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 57 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 02:02:06.553828 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ff8043f-f06a-41f8-b7cc-252da271daa4 admin admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:02:06.569716 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ff8043f-f06a-41f8-b7cc-252da271daa4 admin admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:02:06.575278 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46da2a69-6070-472c-b62c-66b8235f4c24 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:02:06.595241 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46da2a69-6070-472c-b62c-66b8235f4c24 admin admin] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:02:06.595241 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46da2a69-6070-472c-b62c-66b8235f4c24 admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:02:06.595241 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46da2a69-6070-472c-b62c-66b8235f4c24 admin admin] PATH_INFO: `/v3/projects` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:02:06.595241 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46da2a69-6070-472c-b62c-66b8235f4c24 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:02:06.595241 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46da2a69-6070-472c-b62c-66b8235f4c24 admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:02:06.694059 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 852/1689] 10.222.0.27 () {60 vars in 1153 bytes} [Tue Aug 30 02:02:06 2022] POST /identity/v3/projects => generated 381 bytes in 440 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 02:02:06.695384 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdb408e5-9c01-4abd-b73a-dbb72e5a1045 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:02:06.777016 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 838/1690] 10.222.0.27 () {60 vars in 1153 bytes} [Tue Aug 30 02:02:06 2022] POST /identity/v3/projects => generated 367 bytes in 224 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 30 02:02:06.782722 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46da2a69-6070-472c-b62c-66b8235f4c24 admin admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:02:06.891720 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdb408e5-9c01-4abd-b73a-dbb72e5a1045 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:02:06.938913 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46da2a69-6070-472c-b62c-66b8235f4c24 admin admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:02:06.951665 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c69b557b-b103-4f0e-9087-d41d72fe9245 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:02:06.951665 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c69b557b-b103-4f0e-9087-d41d72fe9245 admin admin] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:02:06.951665 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c69b557b-b103-4f0e-9087-d41d72fe9245 admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:02:06.951665 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c69b557b-b103-4f0e-9087-d41d72fe9245 admin admin] PATH_INFO: `/v3/users` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:02:06.952122 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c69b557b-b103-4f0e-9087-d41d72fe9245 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:02:06.952122 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c69b557b-b103-4f0e-9087-d41d72fe9245 admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:02:06.977920 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4606b59-8a9e-4bca-b465-63bcb33d7430 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:02:06.978933 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4606b59-8a9e-4bca-b465-63bcb33d7430 admin admin] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:02:06.979314 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4606b59-8a9e-4bca-b465-63bcb33d7430 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:02:06.979687 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4606b59-8a9e-4bca-b465-63bcb33d7430 admin admin] PATH_INFO: `/v3/users` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:02:06.980460 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4606b59-8a9e-4bca-b465-63bcb33d7430 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:02:06.983728 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4606b59-8a9e-4bca-b465-63bcb33d7430 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:02:07.113014 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 839/1691] 10.222.0.27 () {60 vars in 1144 bytes} [Tue Aug 30 02:02:06 2022] POST /identity/v3/users => generated 284 bytes in 331 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 30 02:02:07.117618 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c69b557b-b103-4f0e-9087-d41d72fe9245 admin admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:02:07.132644 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c69b557b-b103-4f0e-9087-d41d72fe9245 admin admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:02:07.139647 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ea1ac3e-8176-4ad6-a6a8-3735922f6e2d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:02:07.140421 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ea1ac3e-8176-4ad6-a6a8-3735922f6e2d admin admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:02:07.140813 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ea1ac3e-8176-4ad6-a6a8-3735922f6e2d admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:02:07.141186 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ea1ac3e-8176-4ad6-a6a8-3735922f6e2d admin admin] PATH_INFO: `/v3/roles` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:02:07.143463 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ea1ac3e-8176-4ad6-a6a8-3735922f6e2d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:02:07.144724 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ea1ac3e-8176-4ad6-a6a8-3735922f6e2d admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:02:07.209275 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 840/1692] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Aug 30 02:02:07 2022] GET /identity/v3/roles => generated 1420 bytes in 93 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 02:02:07.213207 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ea1ac3e-8176-4ad6-a6a8-3735922f6e2d admin admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:02:07.241641 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ea1ac3e-8176-4ad6-a6a8-3735922f6e2d admin admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:02:07.257365 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36ff0ced-4391-4ab9-ad2d-e674381822d9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:02:07.258030 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36ff0ced-4391-4ab9-ad2d-e674381822d9 admin admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:02:07.258312 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36ff0ced-4391-4ab9-ad2d-e674381822d9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:02:07.258582 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36ff0ced-4391-4ab9-ad2d-e674381822d9 admin admin] PATH_INFO: `/v3/roles` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:02:07.259238 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36ff0ced-4391-4ab9-ad2d-e674381822d9 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:02:07.262036 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36ff0ced-4391-4ab9-ad2d-e674381822d9 admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:02:07.278449 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 853/1693] 10.222.0.27 () {60 vars in 1144 bytes} [Tue Aug 30 02:02:06 2022] POST /identity/v3/users => generated 297 bytes in 583 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 30 02:02:07.283858 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4606b59-8a9e-4bca-b465-63bcb33d7430 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:02:07.296453 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 841/1694] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Aug 30 02:02:07 2022] GET /identity/v3/roles => generated 1420 bytes in 84 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 02:02:07.301005 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36ff0ced-4391-4ab9-ad2d-e674381822d9 admin admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:02:07.310834 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4606b59-8a9e-4bca-b465-63bcb33d7430 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:02:07.326781 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1d04539-17c1-492f-b2cc-10fc108d2a53 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:02:07.326781 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1d04539-17c1-492f-b2cc-10fc108d2a53 admin admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:02:07.326781 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1d04539-17c1-492f-b2cc-10fc108d2a53 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:02:07.326781 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1d04539-17c1-492f-b2cc-10fc108d2a53 admin admin] PATH_INFO: `/v3/roles` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:02:07.340962 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36ff0ced-4391-4ab9-ad2d-e674381822d9 admin admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:02:07.341062 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1d04539-17c1-492f-b2cc-10fc108d2a53 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:02:07.344723 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1d04539-17c1-492f-b2cc-10fc108d2a53 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:02:07.347319 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6adbde4-17f1-4af2-8bf0-d3355f6135fc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:02:07.353101 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6adbde4-17f1-4af2-8bf0-d3355f6135fc admin admin] REQUEST_METHOD: `PUT` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:02:07.353864 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6adbde4-17f1-4af2-8bf0-d3355f6135fc admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:02:07.355207 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6adbde4-17f1-4af2-8bf0-d3355f6135fc admin admin] PATH_INFO: `/v3/projects/34e6a886fdef4899a6261200d9ecd640/users/3e63fec0ec7649aa86422d775a018a49/roles/1fb81afcb9f245f3b989add94598a063` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:02:07.370890 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 854/1695] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Aug 30 02:02:07 2022] GET /identity/v3/roles => generated 1420 bytes in 87 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 02:02:07.376298 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1d04539-17c1-492f-b2cc-10fc108d2a53 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:02:07.401494 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6adbde4-17f1-4af2-8bf0-d3355f6135fc admin admin] RBAC: Authorizing `identity:create_grant(project_id=34e6a886fdef4899a6261200d9ecd640, user_id=3e63fec0ec7649aa86422d775a018a49, role_id=1fb81afcb9f245f3b989add94598a063)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:02:07.404873 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6adbde4-17f1-4af2-8bf0-d3355f6135fc admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:02:07.411734 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1d04539-17c1-492f-b2cc-10fc108d2a53 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:02:07.431247 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06e716eb-67ac-40c3-9aeb-ac17fd540d03 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:02:07.431938 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06e716eb-67ac-40c3-9aeb-ac17fd540d03 admin admin] REQUEST_METHOD: `PUT` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:02:07.432221 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06e716eb-67ac-40c3-9aeb-ac17fd540d03 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:02:07.432499 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06e716eb-67ac-40c3-9aeb-ac17fd540d03 admin admin] PATH_INFO: `/v3/projects/b5e05350686f48a0a396f6f415734c9c/users/d01a8d9693e449f0a8567e352935e231/roles/ead5fac86a0747ea8823a82efcea8ed7` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:02:07.475513 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06e716eb-67ac-40c3-9aeb-ac17fd540d03 admin admin] RBAC: Authorizing `identity:create_grant(project_id=b5e05350686f48a0a396f6f415734c9c, user_id=d01a8d9693e449f0a8567e352935e231, role_id=ead5fac86a0747ea8823a82efcea8ed7)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:02:07.478806 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06e716eb-67ac-40c3-9aeb-ac17fd540d03 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:02:07.539925 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 842/1696] 10.222.0.27 () {60 vars in 1483 bytes} [Tue Aug 30 02:02:07 2022] PUT /identity/v3/projects/34e6a886fdef4899a6261200d9ecd640/users/3e63fec0ec7649aa86422d775a018a49/roles/1fb81afcb9f245f3b989add94598a063 => generated 0 bytes in 240 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 30 02:02:07.552827 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b94a9984-575c-4e6e-9dff-2222b1cfb5d5 None None] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:02:07.553121 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b94a9984-575c-4e6e-9dff-2222b1cfb5d5 None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:02:07.553397 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b94a9984-575c-4e6e-9dff-2222b1cfb5d5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:02:07.581381 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 855/1697] 10.222.0.27 () {60 vars in 1483 bytes} [Tue Aug 30 02:02:07 2022] PUT /identity/v3/projects/b5e05350686f48a0a396f6f415734c9c/users/d01a8d9693e449f0a8567e352935e231/roles/ead5fac86a0747ea8823a82efcea8ed7 => generated 0 bytes in 206 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 02:02:07.586266 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06e716eb-67ac-40c3-9aeb-ac17fd540d03 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:02:07.619526 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.auth.core [None req-b94a9984-575c-4e6e-9dff-2222b1cfb5d5 None None] MFA Rules not processed for user `3e63fec0ec7649aa86422d775a018a49`. Rule list: `[]` (Enabled: `True`). {{(pid=81578) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:02:07.621803 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-b94a9984-575c-4e6e-9dff-2222b1cfb5d5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:02:07.697091 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06e716eb-67ac-40c3-9aeb-ac17fd540d03 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:02:07.703176 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8020070-321b-4f57-9dce-5fc6108e77ed admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:02:07.703967 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8020070-321b-4f57-9dce-5fc6108e77ed admin admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:02:07.704957 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8020070-321b-4f57-9dce-5fc6108e77ed admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:02:07.704957 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8020070-321b-4f57-9dce-5fc6108e77ed admin admin] PATH_INFO: `/v3/roles` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:02:07.707964 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8020070-321b-4f57-9dce-5fc6108e77ed admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:02:07.710766 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8020070-321b-4f57-9dce-5fc6108e77ed admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:02:07.725263 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 856/1698] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Aug 30 02:02:07 2022] GET /identity/v3/roles => generated 1420 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 02:02:07.726739 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16535e21-f226-41b7-9ba1-5c0780068101 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:02:07.727240 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16535e21-f226-41b7-9ba1-5c0780068101 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:02:07.727763 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16535e21-f226-41b7-9ba1-5c0780068101 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:02:07.776044 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 843/1699] 10.222.0.27 () {58 vars in 958 bytes} [Tue Aug 30 02:02:07 2022] POST /identity/v3/auth/tokens => generated 3244 bytes in 225 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 30 02:02:07.777003 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-16535e21-f226-41b7-9ba1-5c0780068101 None None] MFA Rules not processed for user `44f9cfa0ebb84922a9af96ba030ee57a`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:02:07.779067 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-16535e21-f226-41b7-9ba1-5c0780068101 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:02:07.779340 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b94a9984-575c-4e6e-9dff-2222b1cfb5d5 None None] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:02:07.804905 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b94a9984-575c-4e6e-9dff-2222b1cfb5d5 None None] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:02:07.812616 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e9626c6-1b1a-428f-bf5e-e39e99f84ba4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:02:07.813715 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e9626c6-1b1a-428f-bf5e-e39e99f84ba4 admin admin] REQUEST_METHOD: `PUT` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:02:07.813989 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e9626c6-1b1a-428f-bf5e-e39e99f84ba4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:02:07.815770 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e9626c6-1b1a-428f-bf5e-e39e99f84ba4 admin admin] PATH_INFO: `/v3/projects/b5e05350686f48a0a396f6f415734c9c/users/d01a8d9693e449f0a8567e352935e231/roles/ead5fac86a0747ea8823a82efcea8ed7` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:02:07.818054 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e9626c6-1b1a-428f-bf5e-e39e99f84ba4 admin admin] RBAC: Authorizing `identity:create_grant(project_id=b5e05350686f48a0a396f6f415734c9c, user_id=d01a8d9693e449f0a8567e352935e231, role_id=ead5fac86a0747ea8823a82efcea8ed7)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:02:07.821929 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e9626c6-1b1a-428f-bf5e-e39e99f84ba4 admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:02:07.850738 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 844/1700] 10.222.0.27 () {60 vars in 1483 bytes} [Tue Aug 30 02:02:07 2022] PUT /identity/v3/projects/b5e05350686f48a0a396f6f415734c9c/users/d01a8d9693e449f0a8567e352935e231/roles/ead5fac86a0747ea8823a82efcea8ed7 => generated 0 bytes in 74 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 02:02:07.851966 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e9626c6-1b1a-428f-bf5e-e39e99f84ba4 admin admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:02:07.913576 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 857/1701] 10.222.0.27 () {58 vars in 958 bytes} [Tue Aug 30 02:02:07 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 188 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 02:02:07.914880 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f664929-960f-400d-a091-0257f70b7840 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:02:07.915165 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f664929-960f-400d-a091-0257f70b7840 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:02:07.915437 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f664929-960f-400d-a091-0257f70b7840 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:02:07.952365 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-9f664929-960f-400d-a091-0257f70b7840 None None] MFA Rules not processed for user `d01a8d9693e449f0a8567e352935e231`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:02:07.954563 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-9f664929-960f-400d-a091-0257f70b7840 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:02:07.985441 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e9626c6-1b1a-428f-bf5e-e39e99f84ba4 admin admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:02:07.991186 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b7456e5-4053-4664-815b-9921ace981d7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:02:07.992096 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b7456e5-4053-4664-815b-9921ace981d7 admin admin] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:02:07.992573 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b7456e5-4053-4664-815b-9921ace981d7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:02:07.993077 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b7456e5-4053-4664-815b-9921ace981d7 admin admin] PATH_INFO: `/v3/projects` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:02:07.994467 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b7456e5-4053-4664-815b-9921ace981d7 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:02:07.997079 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b7456e5-4053-4664-815b-9921ace981d7 admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:02:08.098918 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 858/1702] 10.222.0.27 () {58 vars in 958 bytes} [Tue Aug 30 02:02:07 2022] POST /identity/v3/auth/tokens => generated 3325 bytes in 185 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 02:02:08.100353 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b35fe87c-0a90-478c-b66b-cf83a4f008bf None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:02:08.100643 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b35fe87c-0a90-478c-b66b-cf83a4f008bf None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:02:08.100916 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b35fe87c-0a90-478c-b66b-cf83a4f008bf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:02:08.148739 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-b35fe87c-0a90-478c-b66b-cf83a4f008bf None None] MFA Rules not processed for user `44f9cfa0ebb84922a9af96ba030ee57a`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:02:08.153595 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-b35fe87c-0a90-478c-b66b-cf83a4f008bf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:02:08.256701 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 845/1703] 10.222.0.27 () {60 vars in 1153 bytes} [Tue Aug 30 02:02:07 2022] POST /identity/v3/projects => generated 367 bytes in 387 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 02:02:08.256701 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b7456e5-4053-4664-815b-9921ace981d7 admin admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:02:08.256701 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-6b7456e5-4053-4664-815b-9921ace981d7 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:02:08.281355 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:02:08.339023 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 859/1704] 10.222.0.27 () {58 vars in 958 bytes} [Tue Aug 30 02:02:08 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 240 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 02:02:08.340506 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b35fe87c-0a90-478c-b66b-cf83a4f008bf None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:02:08.358595 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b35fe87c-0a90-478c-b66b-cf83a4f008bf None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:02:08.386571 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b7456e5-4053-4664-815b-9921ace981d7 admin admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:02:08.394635 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce864ef2-5653-40b7-874d-20c8ac5c1d57 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ab936b3767a8417aa80c8c8f1b84b667', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4562f3951fbe465280073e60a11e44e2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:02:08.395575 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce864ef2-5653-40b7-874d-20c8ac5c1d57 service cinder] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:02:08.397159 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce864ef2-5653-40b7-874d-20c8ac5c1d57 service cinder] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:02:08.397572 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce864ef2-5653-40b7-874d-20c8ac5c1d57 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:02:08.424179 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=93,keystone:INSERT=4,keystone:DELETE=3 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:02:08.486876 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf030d85-8b51-4554-aa44-6827813a8e45 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:02:08.487667 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf030d85-8b51-4554-aa44-6827813a8e45 admin admin] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:02:08.487954 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf030d85-8b51-4554-aa44-6827813a8e45 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:02:08.488229 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf030d85-8b51-4554-aa44-6827813a8e45 admin admin] PATH_INFO: `/v3/users` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:02:08.489041 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf030d85-8b51-4554-aa44-6827813a8e45 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:02:08.495991 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf030d85-8b51-4554-aa44-6827813a8e45 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:02:08.553615 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce864ef2-5653-40b7-874d-20c8ac5c1d57 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:02:08.555294 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce864ef2-5653-40b7-874d-20c8ac5c1d57 service cinder] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:02:08.578353 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 846/1705] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Aug 30 02:02:08 2022] GET /identity/v3/auth/tokens => generated 3325 bytes in 339 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 02:02:08.582522 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce864ef2-5653-40b7-874d-20c8ac5c1d57 service cinder] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:02:08.591100 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-ce864ef2-5653-40b7-874d-20c8ac5c1d57 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:02:08.639902 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 860/1706] 10.222.0.27 () {60 vars in 1144 bytes} [Tue Aug 30 02:02:08 2022] POST /identity/v3/users => generated 290 bytes in 300 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 02:02:08.643605 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf030d85-8b51-4554-aa44-6827813a8e45 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:02:08.666862 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf030d85-8b51-4554-aa44-6827813a8e45 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:02:08.679158 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a3ede46-c766-43c0-a685-78a5f6171964 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:02:08.680041 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a3ede46-c766-43c0-a685-78a5f6171964 admin admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:02:08.680419 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a3ede46-c766-43c0-a685-78a5f6171964 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:02:08.680792 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a3ede46-c766-43c0-a685-78a5f6171964 admin admin] PATH_INFO: `/v3/roles` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:02:08.681914 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a3ede46-c766-43c0-a685-78a5f6171964 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:02:08.686356 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a3ede46-c766-43c0-a685-78a5f6171964 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:02:08.728458 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 861/1707] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Aug 30 02:02:08 2022] GET /identity/v3/roles => generated 1420 bytes in 85 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 02:02:08.732972 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a3ede46-c766-43c0-a685-78a5f6171964 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:02:08.738175 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce864ef2-5653-40b7-874d-20c8ac5c1d57 service cinder] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:02:08.753597 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16d3413e-5a2c-4954-9637-b58d8deba2fd service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd64f2e71361644589bec8a06e6eceed1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4562f3951fbe465280073e60a11e44e2', 'project_domain_id': 'default', 'roles': ['member', 'service', 'admin', 'ResellerAdmin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:02:08.755945 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a3ede46-c766-43c0-a685-78a5f6171964 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:02:08.756192 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16d3413e-5a2c-4954-9637-b58d8deba2fd service nova] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:02:08.756713 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16d3413e-5a2c-4954-9637-b58d8deba2fd service nova] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:02:08.757164 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16d3413e-5a2c-4954-9637-b58d8deba2fd service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:02:08.767208 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65d47a07-9c17-4ac4-aaa1-5d38072deb73 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:02:08.767964 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65d47a07-9c17-4ac4-aaa1-5d38072deb73 admin admin] REQUEST_METHOD: `PUT` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:02:08.768311 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65d47a07-9c17-4ac4-aaa1-5d38072deb73 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:02:08.768649 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65d47a07-9c17-4ac4-aaa1-5d38072deb73 admin admin] PATH_INFO: `/v3/projects/2d2e26f8cce64a02929ead6ed540d36d/users/ca58bd7727d44d3594694c951ed88bcd/roles/ead5fac86a0747ea8823a82efcea8ed7` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:02:08.790035 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65d47a07-9c17-4ac4-aaa1-5d38072deb73 admin admin] RBAC: Authorizing `identity:create_grant(project_id=2d2e26f8cce64a02929ead6ed540d36d, user_id=ca58bd7727d44d3594694c951ed88bcd, role_id=ead5fac86a0747ea8823a82efcea8ed7)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:02:08.792883 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65d47a07-9c17-4ac4-aaa1-5d38072deb73 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:02:08.881081 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16d3413e-5a2c-4954-9637-b58d8deba2fd service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:02:08.883316 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16d3413e-5a2c-4954-9637-b58d8deba2fd service nova] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:02:08.895192 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 862/1708] 10.222.0.27 () {60 vars in 1483 bytes} [Tue Aug 30 02:02:08 2022] PUT /identity/v3/projects/2d2e26f8cce64a02929ead6ed540d36d/users/ca58bd7727d44d3594694c951ed88bcd/roles/ead5fac86a0747ea8823a82efcea8ed7 => generated 0 bytes in 163 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 30 02:02:08.899685 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65d47a07-9c17-4ac4-aaa1-5d38072deb73 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:02:08.914818 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 847/1709] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Aug 30 02:02:08 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 336 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 02:02:09.073801 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65d47a07-9c17-4ac4-aaa1-5d38072deb73 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:02:09.094250 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b963c2e4-a088-4f7a-9c41-4becfbbd4f92 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:02:09.095392 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b963c2e4-a088-4f7a-9c41-4becfbbd4f92 admin admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:02:09.095392 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b963c2e4-a088-4f7a-9c41-4becfbbd4f92 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:02:09.095751 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b963c2e4-a088-4f7a-9c41-4becfbbd4f92 admin admin] PATH_INFO: `/v3/roles` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:02:09.096626 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b963c2e4-a088-4f7a-9c41-4becfbbd4f92 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:02:09.098382 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b963c2e4-a088-4f7a-9c41-4becfbbd4f92 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:02:09.123179 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16d3413e-5a2c-4954-9637-b58d8deba2fd service nova] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:02:09.130476 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 863/1710] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Aug 30 02:02:08 2022] GET /identity/v3/roles => generated 1420 bytes in 230 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 02:02:09.134908 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16d3413e-5a2c-4954-9637-b58d8deba2fd service nova] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:02:09.142206 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-371a2eb6-672b-4477-b92f-f13a687508c4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:02:09.142967 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-371a2eb6-672b-4477-b92f-f13a687508c4 admin admin] REQUEST_METHOD: `PUT` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:02:09.143356 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-371a2eb6-672b-4477-b92f-f13a687508c4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:02:09.143759 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-371a2eb6-672b-4477-b92f-f13a687508c4 admin admin] PATH_INFO: `/v3/projects/2d2e26f8cce64a02929ead6ed540d36d/users/ca58bd7727d44d3594694c951ed88bcd/roles/ead5fac86a0747ea8823a82efcea8ed7` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:02:09.146085 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-371a2eb6-672b-4477-b92f-f13a687508c4 admin admin] RBAC: Authorizing `identity:create_grant(project_id=2d2e26f8cce64a02929ead6ed540d36d, user_id=ca58bd7727d44d3594694c951ed88bcd, role_id=ead5fac86a0747ea8823a82efcea8ed7)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:02:09.160156 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-371a2eb6-672b-4477-b92f-f13a687508c4 admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:02:09.209445 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69ca17cd-2723-4922-845c-6469901cd6ad None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:02:09.209445 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69ca17cd-2723-4922-845c-6469901cd6ad None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:02:09.209445 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69ca17cd-2723-4922-845c-6469901cd6ad None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:02:09.209805 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 848/1711] 10.222.0.27 () {60 vars in 1483 bytes} [Tue Aug 30 02:02:09 2022] PUT /identity/v3/projects/2d2e26f8cce64a02929ead6ed540d36d/users/ca58bd7727d44d3594694c951ed88bcd/roles/ead5fac86a0747ea8823a82efcea8ed7 => generated 0 bytes in 72 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 02:02:09.246439 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-69ca17cd-2723-4922-845c-6469901cd6ad None None] MFA Rules not processed for user `ca58bd7727d44d3594694c951ed88bcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:02:09.248208 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-69ca17cd-2723-4922-845c-6469901cd6ad None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:02:09.423246 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 864/1712] 10.222.0.27 () {58 vars in 958 bytes} [Tue Aug 30 02:02:09 2022] POST /identity/v3/auth/tokens => generated 3311 bytes in 219 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 02:02:09.494088 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-371a2eb6-672b-4477-b92f-f13a687508c4 admin admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:02:09.496113 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-371a2eb6-672b-4477-b92f-f13a687508c4 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:02:09.668038 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-371a2eb6-672b-4477-b92f-f13a687508c4 admin admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:02:09.678855 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45d31fce-2e15-4e5f-8d12-315ade2f1b1b service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ab936b3767a8417aa80c8c8f1b84b667', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4562f3951fbe465280073e60a11e44e2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:02:09.679652 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45d31fce-2e15-4e5f-8d12-315ade2f1b1b service cinder] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:02:09.680038 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45d31fce-2e15-4e5f-8d12-315ade2f1b1b service cinder] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:02:09.680554 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45d31fce-2e15-4e5f-8d12-315ade2f1b1b service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:02:09.726078 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45d31fce-2e15-4e5f-8d12-315ade2f1b1b service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:02:09.742995 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45d31fce-2e15-4e5f-8d12-315ade2f1b1b service cinder] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:02:09.777978 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 849/1713] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Aug 30 02:02:09 2022] GET /identity/v3/auth/tokens => generated 3311 bytes in 285 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 02:02:09.996178 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:02:10.247247 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f348b46-aec5-47ad-9345-b81dc4b7033b None None] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:02:10.247709 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f348b46-aec5-47ad-9345-b81dc4b7033b None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:02:10.248081 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f348b46-aec5-47ad-9345-b81dc4b7033b None None] PATH_INFO: `/` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:02:10.249957 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 865/1714] 10.222.0.27 () {54 vars in 912 bytes} [Tue Aug 30 02:02:10 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 02:02:10.259265 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fc193ce-0a64-45ac-bf2f-c9e3226c7841 None None] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:02:10.259711 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fc193ce-0a64-45ac-bf2f-c9e3226c7841 None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:02:10.260081 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fc193ce-0a64-45ac-bf2f-c9e3226c7841 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:02:10.350078 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.auth.core [None req-0fc193ce-0a64-45ac-bf2f-c9e3226c7841 None None] MFA Rules not processed for user `f3ea1580c56f4814a92bbb2acf542905`. Rule list: `[]` (Enabled: `True`). {{(pid=81578) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:02:10.351835 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-0fc193ce-0a64-45ac-bf2f-c9e3226c7841 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:02:10.390174 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f348b46-aec5-47ad-9345-b81dc4b7033b None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:02:10.530346 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 850/1715] 10.222.0.27 () {58 vars in 1009 bytes} [Tue Aug 30 02:02:10 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 272 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 30 02:02:10.531730 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fc193ce-0a64-45ac-bf2f-c9e3226c7841 None None] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:02:10.558650 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fc193ce-0a64-45ac-bf2f-c9e3226c7841 None None] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:02:10.573280 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a224ac4-9b2a-437e-a687-23aa7db66dd9 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ab936b3767a8417aa80c8c8f1b84b667', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4562f3951fbe465280073e60a11e44e2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:02:10.574134 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a224ac4-9b2a-437e-a687-23aa7db66dd9 service cinder] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:02:10.574516 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a224ac4-9b2a-437e-a687-23aa7db66dd9 service cinder] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:02:10.574887 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a224ac4-9b2a-437e-a687-23aa7db66dd9 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:02:10.609875 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f348b46-aec5-47ad-9345-b81dc4b7033b None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:02:10.636374 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2de8c3c0-ab9d-4bb6-b29d-327cc717855e tempest-VolumeQuotaClassesTest-1678975674 tempest-VolumeQuotaClassesTest-1678975674-project-admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd01a8d9693e449f0a8567e352935e231', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b5e05350686f48a0a396f6f415734c9c', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:02:10.637742 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2de8c3c0-ab9d-4bb6-b29d-327cc717855e tempest-VolumeQuotaClassesTest-1678975674 tempest-VolumeQuotaClassesTest-1678975674-project-admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:02:10.637742 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2de8c3c0-ab9d-4bb6-b29d-327cc717855e tempest-VolumeQuotaClassesTest-1678975674 tempest-VolumeQuotaClassesTest-1678975674-project-admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:02:10.646863 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2de8c3c0-ab9d-4bb6-b29d-327cc717855e tempest-VolumeQuotaClassesTest-1678975674 tempest-VolumeQuotaClassesTest-1678975674-project-admin] PATH_INFO: `/v3/domains` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:02:10.655847 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2de8c3c0-ab9d-4bb6-b29d-327cc717855e tempest-VolumeQuotaClassesTest-1678975674 tempest-VolumeQuotaClassesTest-1678975674-project-admin] RBAC: Adding query filter params (name=Default) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 02:02:10.655847 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2de8c3c0-ab9d-4bb6-b29d-327cc717855e tempest-VolumeQuotaClassesTest-1678975674 tempest-VolumeQuotaClassesTest-1678975674-project-admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:02:10.655847 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2de8c3c0-ab9d-4bb6-b29d-327cc717855e tempest-VolumeQuotaClassesTest-1678975674 tempest-VolumeQuotaClassesTest-1678975674-project-admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:02:10.703268 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 866/1716] 10.222.0.27 () {58 vars in 1153 bytes} [Tue Aug 30 02:02:10 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 314 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 30 02:02:10.706425 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2de8c3c0-ab9d-4bb6-b29d-327cc717855e tempest-VolumeQuotaClassesTest-1678975674 tempest-VolumeQuotaClassesTest-1678975674-project-admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:02:10.734963 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2de8c3c0-ab9d-4bb6-b29d-327cc717855e tempest-VolumeQuotaClassesTest-1678975674 tempest-VolumeQuotaClassesTest-1678975674-project-admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:02:10.749629 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2893b905-66cc-49d6-8950-5802afa2926d tempest-VolumeQuotaClassesTest-1678975674 tempest-VolumeQuotaClassesTest-1678975674-project-admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd01a8d9693e449f0a8567e352935e231', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b5e05350686f48a0a396f6f415734c9c', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:02:10.750364 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2893b905-66cc-49d6-8950-5802afa2926d tempest-VolumeQuotaClassesTest-1678975674 tempest-VolumeQuotaClassesTest-1678975674-project-admin] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:02:10.750768 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2893b905-66cc-49d6-8950-5802afa2926d tempest-VolumeQuotaClassesTest-1678975674 tempest-VolumeQuotaClassesTest-1678975674-project-admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:02:10.751147 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2893b905-66cc-49d6-8950-5802afa2926d tempest-VolumeQuotaClassesTest-1678975674 tempest-VolumeQuotaClassesTest-1678975674-project-admin] PATH_INFO: `/v3/projects` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:02:10.753034 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2893b905-66cc-49d6-8950-5802afa2926d tempest-VolumeQuotaClassesTest-1678975674 tempest-VolumeQuotaClassesTest-1678975674-project-admin] RBAC: Authorizing `identity:create_project()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:02:10.755300 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2893b905-66cc-49d6-8950-5802afa2926d tempest-VolumeQuotaClassesTest-1678975674 tempest-VolumeQuotaClassesTest-1678975674-project-admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:02:10.785519 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a224ac4-9b2a-437e-a687-23aa7db66dd9 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:02:10.789893 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a224ac4-9b2a-437e-a687-23aa7db66dd9 service cinder] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:02:10.808501 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 851/1717] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Aug 30 02:02:10 2022] GET /identity/v3/auth/tokens => generated 3244 bytes in 277 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 02:02:10.819956 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a224ac4-9b2a-437e-a687-23aa7db66dd9 service cinder] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:02:10.821600 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-2a224ac4-9b2a-437e-a687-23aa7db66dd9 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:02:10.840686 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a224ac4-9b2a-437e-a687-23aa7db66dd9 service cinder] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:02:10.844348 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:02:10.851454 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fdc9cfa-82c2-423e-8a76-b524fc5043d5 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f3ea1580c56f4814a92bbb2acf542905', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4562f3951fbe465280073e60a11e44e2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:02:10.852403 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fdc9cfa-82c2-423e-8a76-b524fc5043d5 service neutron] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:02:10.852787 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fdc9cfa-82c2-423e-8a76-b524fc5043d5 service neutron] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:02:10.853165 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fdc9cfa-82c2-423e-8a76-b524fc5043d5 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:02:10.870523 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fdc9cfa-82c2-423e-8a76-b524fc5043d5 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:02:10.876473 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fdc9cfa-82c2-423e-8a76-b524fc5043d5 service neutron] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:02:10.896322 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 852/1718] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Aug 30 02:02:10 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 76 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 02:02:10.982842 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 867/1719] 10.222.0.27 () {60 vars in 1153 bytes} [Tue Aug 30 02:02:10 2022] POST /identity/v3/projects => generated 354 bytes in 274 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 02:02:10.984635 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fdc9cfa-82c2-423e-8a76-b524fc5043d5 service neutron] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:02:10.998699 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fdc9cfa-82c2-423e-8a76-b524fc5043d5 service neutron] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:02:11.018659 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=93,keystone:DELETE=4,keystone:INSERT=3 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:02:11.125807 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59537059-ed1b-4b01-804f-0700f89813ef tempest-VolumeQuotaClassesTest-1678975674 tempest-VolumeQuotaClassesTest-1678975674-project-admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd01a8d9693e449f0a8567e352935e231', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b5e05350686f48a0a396f6f415734c9c', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:02:11.128930 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59537059-ed1b-4b01-804f-0700f89813ef tempest-VolumeQuotaClassesTest-1678975674 tempest-VolumeQuotaClassesTest-1678975674-project-admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:02:11.128930 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59537059-ed1b-4b01-804f-0700f89813ef tempest-VolumeQuotaClassesTest-1678975674 tempest-VolumeQuotaClassesTest-1678975674-project-admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:02:11.128930 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59537059-ed1b-4b01-804f-0700f89813ef tempest-VolumeQuotaClassesTest-1678975674 tempest-VolumeQuotaClassesTest-1678975674-project-admin] PATH_INFO: `/v3/domains` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:02:11.128930 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59537059-ed1b-4b01-804f-0700f89813ef tempest-VolumeQuotaClassesTest-1678975674 tempest-VolumeQuotaClassesTest-1678975674-project-admin] RBAC: Adding query filter params (name=Default) {{(pid=81578) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 02:02:11.128930 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59537059-ed1b-4b01-804f-0700f89813ef tempest-VolumeQuotaClassesTest-1678975674 tempest-VolumeQuotaClassesTest-1678975674-project-admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:02:11.130383 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59537059-ed1b-4b01-804f-0700f89813ef tempest-VolumeQuotaClassesTest-1678975674 tempest-VolumeQuotaClassesTest-1678975674-project-admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:02:11.148122 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 853/1720] 10.222.0.27 () {58 vars in 1153 bytes} [Tue Aug 30 02:02:10 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 02:02:11.488430 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7380622f-21a5-4840-9598-b9fe6630b05d None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:02:11.489016 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7380622f-21a5-4840-9598-b9fe6630b05d None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:02:11.489552 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7380622f-21a5-4840-9598-b9fe6630b05d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:02:11.527960 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-7380622f-21a5-4840-9598-b9fe6630b05d None None] MFA Rules not processed for user `5e7584fe83684b01baa31ab67edac026`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:02:11.530590 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-7380622f-21a5-4840-9598-b9fe6630b05d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:02:11.640724 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38707e3f-5604-4ff8-996f-53a1c116ff99 None None] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:02:11.641013 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38707e3f-5604-4ff8-996f-53a1c116ff99 None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:02:11.641119 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 868/1721] 10.222.0.27 () {58 vars in 958 bytes} [Tue Aug 30 02:02:11 2022] POST /identity/v3/auth/tokens => generated 3268 bytes in 154 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 02:02:11.641848 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38707e3f-5604-4ff8-996f-53a1c116ff99 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:02:11.683832 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.auth.core [None req-38707e3f-5604-4ff8-996f-53a1c116ff99 None None] MFA Rules not processed for user `5e7584fe83684b01baa31ab67edac026`. Rule list: `[]` (Enabled: `True`). {{(pid=81578) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:02:11.685358 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-38707e3f-5604-4ff8-996f-53a1c116ff99 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:02:11.710853 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 854/1722] 10.222.0.27 () {58 vars in 958 bytes} [Tue Aug 30 02:02:11 2022] POST /identity/v3/auth/tokens => generated 3268 bytes in 72 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 30 02:02:11.722762 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7380622f-21a5-4840-9598-b9fe6630b05d None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:02:11.724534 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-7380622f-21a5-4840-9598-b9fe6630b05d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:02:11.945618 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38707e3f-5604-4ff8-996f-53a1c116ff99 None None] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:02:11.946571 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7380622f-21a5-4840-9598-b9fe6630b05d None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:02:11.957813 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-088757e4-44d9-4d5b-a75a-4546348114a1 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd64f2e71361644589bec8a06e6eceed1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4562f3951fbe465280073e60a11e44e2', 'project_domain_id': 'default', 'roles': ['member', 'service', 'admin', 'ResellerAdmin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:02:11.958630 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-088757e4-44d9-4d5b-a75a-4546348114a1 service nova] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:02:11.961017 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-088757e4-44d9-4d5b-a75a-4546348114a1 service nova] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:02:11.961017 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-088757e4-44d9-4d5b-a75a-4546348114a1 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:02:11.964976 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38707e3f-5604-4ff8-996f-53a1c116ff99 None None] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:02:11.971459 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-088757e4-44d9-4d5b-a75a-4546348114a1 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:02:11.973148 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-088757e4-44d9-4d5b-a75a-4546348114a1 service nova] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:02:11.974389 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99101bb0-3ede-4aff-85fd-5c6c4d65a153 tempest-VolumeQuotaClassesTest-1678975674 tempest-VolumeQuotaClassesTest-1678975674-project-admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd01a8d9693e449f0a8567e352935e231', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b5e05350686f48a0a396f6f415734c9c', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:02:11.975319 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99101bb0-3ede-4aff-85fd-5c6c4d65a153 tempest-VolumeQuotaClassesTest-1678975674 tempest-VolumeQuotaClassesTest-1678975674-project-admin] REQUEST_METHOD: `DELETE` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:02:11.975741 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99101bb0-3ede-4aff-85fd-5c6c4d65a153 tempest-VolumeQuotaClassesTest-1678975674 tempest-VolumeQuotaClassesTest-1678975674-project-admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:02:11.976168 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99101bb0-3ede-4aff-85fd-5c6c4d65a153 tempest-VolumeQuotaClassesTest-1678975674 tempest-VolumeQuotaClassesTest-1678975674-project-admin] PATH_INFO: `/v3/projects/c5b26d01218d4861996503ea7818f282` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:02:11.977803 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99101bb0-3ede-4aff-85fd-5c6c4d65a153 tempest-VolumeQuotaClassesTest-1678975674 tempest-VolumeQuotaClassesTest-1678975674-project-admin] RBAC: Authorizing `identity:delete_project(project_id=c5b26d01218d4861996503ea7818f282)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:02:11.980132 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99101bb0-3ede-4aff-85fd-5c6c4d65a153 tempest-VolumeQuotaClassesTest-1678975674 tempest-VolumeQuotaClassesTest-1678975674-project-admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:02:12.003885 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 869/1723] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Aug 30 02:02:11 2022] GET /identity/v3/auth/tokens => generated 3268 bytes in 277 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 02:02:12.296227 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-99101bb0-3ede-4aff-85fd-5c6c4d65a153 tempest-VolumeQuotaClassesTest-1678975674 tempest-VolumeQuotaClassesTest-1678975674-project-admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'c5b26d01218d4861996503ea7818f282', 'request_id': 'req-99101bb0-3ede-4aff-85fd-5c6c4d65a153'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:02:12.315426 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-99101bb0-3ede-4aff-85fd-5c6c4d65a153 tempest-VolumeQuotaClassesTest-1678975674 tempest-VolumeQuotaClassesTest-1678975674-project-admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'c5b26d01218d4861996503ea7818f282', 'request_id': 'req-99101bb0-3ede-4aff-85fd-5c6c4d65a153'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:02:12.348405 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-99101bb0-3ede-4aff-85fd-5c6c4d65a153 tempest-VolumeQuotaClassesTest-1678975674 tempest-VolumeQuotaClassesTest-1678975674-project-admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'c5b26d01218d4861996503ea7818f282', 'request_id': 'req-99101bb0-3ede-4aff-85fd-5c6c4d65a153'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:02:12.413518 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-088757e4-44d9-4d5b-a75a-4546348114a1 service nova] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:02:12.415641 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-088757e4-44d9-4d5b-a75a-4546348114a1 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:02:12.574841 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-088757e4-44d9-4d5b-a75a-4546348114a1 service nova] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:02:12.631449 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-99101bb0-3ede-4aff-85fd-5c6c4d65a153 tempest-VolumeQuotaClassesTest-1678975674 tempest-VolumeQuotaClassesTest-1678975674-project-admin] The token cache is being invalidate because project c5b26d01218d4861996503ea7818f282 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81578) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 30 02:02:12.631821 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-99101bb0-3ede-4aff-85fd-5c6c4d65a153 tempest-VolumeQuotaClassesTest-1678975674 tempest-VolumeQuotaClassesTest-1678975674-project-admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:02:12.632768 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-99101bb0-3ede-4aff-85fd-5c6c4d65a153 tempest-VolumeQuotaClassesTest-1678975674 tempest-VolumeQuotaClassesTest-1678975674-project-admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:02:12.635399 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 855/1724] 10.222.0.27 () {58 vars in 1233 bytes} [Tue Aug 30 02:02:11 2022] DELETE /identity/v3/projects/c5b26d01218d4861996503ea7818f282 => generated 0 bytes in 690 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 30 02:02:12.682286 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a628ab42-5046-49bb-a437-9d8327e6c660 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '379536ff06fd4f719899e5dde732e63d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4562f3951fbe465280073e60a11e44e2', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:02:12.683165 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a628ab42-5046-49bb-a437-9d8327e6c660 service placement] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:02:12.683498 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a628ab42-5046-49bb-a437-9d8327e6c660 service placement] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:02:12.683833 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a628ab42-5046-49bb-a437-9d8327e6c660 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:02:12.687656 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-a628ab42-5046-49bb-a437-9d8327e6c660 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:02:12.706112 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a628ab42-5046-49bb-a437-9d8327e6c660 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:02:12.706112 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a628ab42-5046-49bb-a437-9d8327e6c660 service placement] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:02:12.721118 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 870/1725] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Aug 30 02:02:12 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 307 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 02:02:13.060608 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee7f5441-0a7c-49eb-8a6e-60878a21004c None None] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:02:13.061201 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee7f5441-0a7c-49eb-8a6e-60878a21004c None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:02:13.061344 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee7f5441-0a7c-49eb-8a6e-60878a21004c None None] PATH_INFO: `/` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:02:13.064397 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 856/1726] 10.222.0.27 () {54 vars in 912 bytes} [Tue Aug 30 02:02:13 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 02:02:13.089853 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fd4952f-232f-4424-adce-54f577704281 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:02:13.090151 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fd4952f-232f-4424-adce-54f577704281 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:02:13.090429 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fd4952f-232f-4424-adce-54f577704281 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:02:13.191066 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-2fd4952f-232f-4424-adce-54f577704281 None None] MFA Rules not processed for user `f3ea1580c56f4814a92bbb2acf542905`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:02:13.191066 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-2fd4952f-232f-4424-adce-54f577704281 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:02:13.361492 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 871/1727] 10.222.0.27 () {58 vars in 1009 bytes} [Tue Aug 30 02:02:13 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 273 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 02:02:13.378445 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee7f5441-0a7c-49eb-8a6e-60878a21004c None None] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:02:13.380844 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-ee7f5441-0a7c-49eb-8a6e-60878a21004c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:02:13.437811 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee7f5441-0a7c-49eb-8a6e-60878a21004c None None] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:02:13.443829 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2af8776-b01f-4bd0-94b1-3ecd10ad726c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f3ea1580c56f4814a92bbb2acf542905', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4562f3951fbe465280073e60a11e44e2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:02:13.444485 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2af8776-b01f-4bd0-94b1-3ecd10ad726c service neutron] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:02:13.445183 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2af8776-b01f-4bd0-94b1-3ecd10ad726c service neutron] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:02:13.445636 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2af8776-b01f-4bd0-94b1-3ecd10ad726c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:02:13.456881 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2af8776-b01f-4bd0-94b1-3ecd10ad726c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:02:13.458937 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2af8776-b01f-4bd0-94b1-3ecd10ad726c service neutron] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:02:13.499137 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 857/1728] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Aug 30 02:02:13 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 120 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 02:02:13.801833 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=94,keystone:INSERT=6 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:02:14.464926 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fd4952f-232f-4424-adce-54f577704281 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:02:14.466617 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-2fd4952f-232f-4424-adce-54f577704281 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:02:14.678654 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fd4952f-232f-4424-adce-54f577704281 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:02:14.694443 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-994c379a-8b06-48f2-bcd7-dab69fcfe688 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:02:14.695338 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-994c379a-8b06-48f2-bcd7-dab69fcfe688 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:02:14.695806 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-994c379a-8b06-48f2-bcd7-dab69fcfe688 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:02:14.696252 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-994c379a-8b06-48f2-bcd7-dab69fcfe688 admin admin] PATH_INFO: `/v3/users/43bdd0c830fd448db3df8b88df1281f2` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:02:14.697956 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-994c379a-8b06-48f2-bcd7-dab69fcfe688 admin admin] RBAC: Authorizing `identity:delete_user(user_id=43bdd0c830fd448db3df8b88df1281f2)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:02:14.700158 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-994c379a-8b06-48f2-bcd7-dab69fcfe688 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:02:14.719513 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8bf48ec-d0af-409d-9a51-b67d6ced1588 None None] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:02:14.722666 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8bf48ec-d0af-409d-9a51-b67d6ced1588 None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:02:14.722666 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8bf48ec-d0af-409d-9a51-b67d6ced1588 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:02:14.868361 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.auth.core [None req-b8bf48ec-d0af-409d-9a51-b67d6ced1588 None None] MFA Rules not processed for user `bda980735a1f4948ba695e6c2788f3b4`. Rule list: `[]` (Enabled: `True`). {{(pid=81578) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:02:14.870077 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-b8bf48ec-d0af-409d-9a51-b67d6ced1588 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:02:14.996940 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 858/1729] 10.222.0.27 () {58 vars in 958 bytes} [Tue Aug 30 02:02:14 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 279 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 02:02:15.001827 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-644f82e6-dba1-4946-9b0c-85318351d634 None None] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:02:15.002142 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-644f82e6-dba1-4946-9b0c-85318351d634 None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:02:15.011393 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-644f82e6-dba1-4946-9b0c-85318351d634 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:02:15.085909 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-994c379a-8b06-48f2-bcd7-dab69fcfe688 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '43bdd0c830fd448db3df8b88df1281f2'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:02:15.092617 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-994c379a-8b06-48f2-bcd7-dab69fcfe688 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '43bdd0c830fd448db3df8b88df1281f2'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:02:15.093555 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-994c379a-8b06-48f2-bcd7-dab69fcfe688 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '43bdd0c830fd448db3df8b88df1281f2'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:02:15.099607 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.auth.core [None req-644f82e6-dba1-4946-9b0c-85318351d634 None None] MFA Rules not processed for user `7af1c1123b2e458b84248865287bd00e`. Rule list: `[]` (Enabled: `True`). {{(pid=81578) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:02:15.102997 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-644f82e6-dba1-4946-9b0c-85318351d634 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:02:15.134256 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-994c379a-8b06-48f2-bcd7-dab69fcfe688 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '43bdd0c830fd448db3df8b88df1281f2'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:02:15.165907 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-994c379a-8b06-48f2-bcd7-dab69fcfe688 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '43bdd0c830fd448db3df8b88df1281f2'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:02:15.305030 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 859/1730] 10.222.0.27 () {58 vars in 958 bytes} [Tue Aug 30 02:02:14 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 307 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 02:02:15.306148 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-644f82e6-dba1-4946-9b0c-85318351d634 None None] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:02:15.311875 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-644f82e6-dba1-4946-9b0c-85318351d634 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:02:15.369126 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 872/1731] 10.222.0.27 () {58 vars in 1224 bytes} [Tue Aug 30 02:02:14 2022] DELETE /identity/v3/users/43bdd0c830fd448db3df8b88df1281f2 => generated 0 bytes in 904 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 02:02:15.371251 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06e54f21-3d21-403a-8f14-6d1c947b89f3 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:02:15.371721 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06e54f21-3d21-403a-8f14-6d1c947b89f3 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:02:15.372859 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06e54f21-3d21-403a-8f14-6d1c947b89f3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:02:15.405278 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-06e54f21-3d21-403a-8f14-6d1c947b89f3 None None] MFA Rules not processed for user `5e7584fe83684b01baa31ab67edac026`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:02:15.406258 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-06e54f21-3d21-403a-8f14-6d1c947b89f3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:02:15.513782 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-644f82e6-dba1-4946-9b0c-85318351d634 None None] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:02:15.555624 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 873/1732] 10.222.0.27 () {58 vars in 958 bytes} [Tue Aug 30 02:02:15 2022] POST /identity/v3/auth/tokens => generated 3268 bytes in 186 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 02:02:15.562271 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06e54f21-3d21-403a-8f14-6d1c947b89f3 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:02:15.567021 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-06e54f21-3d21-403a-8f14-6d1c947b89f3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:02:15.649561 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-283c371a-198c-4173-ba4f-909256b460d9 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd64f2e71361644589bec8a06e6eceed1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4562f3951fbe465280073e60a11e44e2', 'project_domain_id': 'default', 'roles': ['member', 'service', 'admin', 'ResellerAdmin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:02:15.650441 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-283c371a-198c-4173-ba4f-909256b460d9 service nova] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:02:15.650860 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-283c371a-198c-4173-ba4f-909256b460d9 service nova] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:02:15.651375 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-283c371a-198c-4173-ba4f-909256b460d9 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:02:15.653854 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-283c371a-198c-4173-ba4f-909256b460d9 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:02:15.673078 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:INSERT=3,keystone:SELECT=97 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:02:15.680413 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06e54f21-3d21-403a-8f14-6d1c947b89f3 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:02:15.699219 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c833b5c8-457a-4e03-90d4-ce499176c3c8 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd64f2e71361644589bec8a06e6eceed1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4562f3951fbe465280073e60a11e44e2', 'project_domain_id': 'default', 'roles': ['member', 'service', 'admin', 'ResellerAdmin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:02:15.699856 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c833b5c8-457a-4e03-90d4-ce499176c3c8 service nova] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:02:15.700618 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c833b5c8-457a-4e03-90d4-ce499176c3c8 service nova] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:02:15.701394 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c833b5c8-457a-4e03-90d4-ce499176c3c8 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:02:15.801371 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-283c371a-198c-4173-ba4f-909256b460d9 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:02:15.808151 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-283c371a-198c-4173-ba4f-909256b460d9 service nova] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:02:15.815946 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c833b5c8-457a-4e03-90d4-ce499176c3c8 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:02:15.817786 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c833b5c8-457a-4e03-90d4-ce499176c3c8 service nova] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:02:15.831953 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 860/1733] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Aug 30 02:02:15 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 526 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 02:02:15.835135 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1510b9f1-00db-4311-92e5-e61890305cd0 None None] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:02:15.835670 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1510b9f1-00db-4311-92e5-e61890305cd0 None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:02:15.837898 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1510b9f1-00db-4311-92e5-e61890305cd0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:02:15.846242 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 874/1734] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Aug 30 02:02:15 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 283 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 02:02:15.883193 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.auth.core [None req-1510b9f1-00db-4311-92e5-e61890305cd0 None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81578) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:02:15.884514 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-1510b9f1-00db-4311-92e5-e61890305cd0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:02:15.968733 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 861/1735] 10.222.0.27 () {58 vars in 958 bytes} [Tue Aug 30 02:02:15 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 136 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 02:02:15.978696 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c833b5c8-457a-4e03-90d4-ce499176c3c8 service nova] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:02:15.980776 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-c833b5c8-457a-4e03-90d4-ce499176c3c8 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:02:16.055187 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c833b5c8-457a-4e03-90d4-ce499176c3c8 service nova] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:02:16.061739 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db6569da-f7b4-4c25-89fc-dd309fb9d575 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f3ea1580c56f4814a92bbb2acf542905', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4562f3951fbe465280073e60a11e44e2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:02:16.062483 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db6569da-f7b4-4c25-89fc-dd309fb9d575 service neutron] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:02:16.062668 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db6569da-f7b4-4c25-89fc-dd309fb9d575 service neutron] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:02:16.062985 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db6569da-f7b4-4c25-89fc-dd309fb9d575 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:02:16.074846 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db6569da-f7b4-4c25-89fc-dd309fb9d575 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:02:16.077309 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db6569da-f7b4-4c25-89fc-dd309fb9d575 service neutron] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:02:16.114179 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 875/1736] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Aug 30 02:02:15 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 136 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 02:02:16.949359 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:02:17.531648 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1510b9f1-00db-4311-92e5-e61890305cd0 None None] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:02:17.534072 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-1510b9f1-00db-4311-92e5-e61890305cd0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:02:17.550912 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1510b9f1-00db-4311-92e5-e61890305cd0 None None] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:02:17.557818 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ffcdaaf-7972-46f4-bf00-a31c39530ee7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:02:17.558465 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ffcdaaf-7972-46f4-bf00-a31c39530ee7 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:02:17.563945 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ffcdaaf-7972-46f4-bf00-a31c39530ee7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:02:17.563945 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ffcdaaf-7972-46f4-bf00-a31c39530ee7 admin admin] PATH_INFO: `/v3/projects/1c2a81204a1541909064de42ae1e95a5` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:02:17.563945 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ffcdaaf-7972-46f4-bf00-a31c39530ee7 admin admin] RBAC: Authorizing `identity:delete_project(project_id=1c2a81204a1541909064de42ae1e95a5)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:02:17.564219 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ffcdaaf-7972-46f4-bf00-a31c39530ee7 admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:02:17.757599 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-2ffcdaaf-7972-46f4-bf00-a31c39530ee7 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '1c2a81204a1541909064de42ae1e95a5', 'request_id': 'req-2ffcdaaf-7972-46f4-bf00-a31c39530ee7'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:02:17.774649 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-2ffcdaaf-7972-46f4-bf00-a31c39530ee7 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '1c2a81204a1541909064de42ae1e95a5', 'request_id': 'req-2ffcdaaf-7972-46f4-bf00-a31c39530ee7'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:02:17.789213 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-2ffcdaaf-7972-46f4-bf00-a31c39530ee7 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '1c2a81204a1541909064de42ae1e95a5', 'request_id': 'req-2ffcdaaf-7972-46f4-bf00-a31c39530ee7'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:02:17.962220 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-2ffcdaaf-7972-46f4-bf00-a31c39530ee7 admin admin] The token cache is being invalidate because project 1c2a81204a1541909064de42ae1e95a5 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81578) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 30 02:02:17.962914 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-2ffcdaaf-7972-46f4-bf00-a31c39530ee7 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:02:17.970443 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-2ffcdaaf-7972-46f4-bf00-a31c39530ee7 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:02:17.973606 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 862/1737] 10.222.0.27 () {58 vars in 1233 bytes} [Tue Aug 30 02:02:17 2022] DELETE /identity/v3/projects/1c2a81204a1541909064de42ae1e95a5 => generated 0 bytes in 439 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 02:02:17.975116 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db6569da-f7b4-4c25-89fc-dd309fb9d575 service neutron] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:02:18.001914 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db6569da-f7b4-4c25-89fc-dd309fb9d575 service neutron] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:02:18.224451 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4930820-5316-492a-944d-c9c7dc4b6847 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:02:18.225115 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4930820-5316-492a-944d-c9c7dc4b6847 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:02:18.232055 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4930820-5316-492a-944d-c9c7dc4b6847 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:02:18.232055 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4930820-5316-492a-944d-c9c7dc4b6847 admin admin] PATH_INFO: `/v3/users/d01a8d9693e449f0a8567e352935e231` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:02:18.232055 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4930820-5316-492a-944d-c9c7dc4b6847 admin admin] RBAC: Authorizing `identity:delete_user(user_id=d01a8d9693e449f0a8567e352935e231)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:02:18.232055 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4930820-5316-492a-944d-c9c7dc4b6847 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:02:18.504109 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:02:18.579821 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-e4930820-5316-492a-944d-c9c7dc4b6847 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'd01a8d9693e449f0a8567e352935e231'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:02:18.580954 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-e4930820-5316-492a-944d-c9c7dc4b6847 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'd01a8d9693e449f0a8567e352935e231'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:02:18.581899 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-e4930820-5316-492a-944d-c9c7dc4b6847 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'd01a8d9693e449f0a8567e352935e231'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:02:18.619531 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-e4930820-5316-492a-944d-c9c7dc4b6847 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'd01a8d9693e449f0a8567e352935e231'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:02:18.628641 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-e4930820-5316-492a-944d-c9c7dc4b6847 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'd01a8d9693e449f0a8567e352935e231'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:02:18.676541 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ffcdaaf-7972-46f4-bf00-a31c39530ee7 admin admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:02:18.678038 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-2ffcdaaf-7972-46f4-bf00-a31c39530ee7 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:02:18.696139 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ffcdaaf-7972-46f4-bf00-a31c39530ee7 admin admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:02:18.717740 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cc875af-1de4-4354-9fbf-0431336cb001 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:02:18.718402 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cc875af-1de4-4354-9fbf-0431336cb001 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:02:18.718682 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cc875af-1de4-4354-9fbf-0431336cb001 admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:02:18.718952 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cc875af-1de4-4354-9fbf-0431336cb001 admin admin] PATH_INFO: `/v3/users/43fc688221d54c3badbb8a6ba3b6d9e3` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:02:18.720452 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1cc875af-1de4-4354-9fbf-0431336cb001 admin admin] RBAC: Authorizing `identity:delete_user(user_id=43fc688221d54c3badbb8a6ba3b6d9e3)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:02:18.739300 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1cc875af-1de4-4354-9fbf-0431336cb001 admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:02:18.795136 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 876/1738] 10.222.0.27 () {58 vars in 1224 bytes} [Tue Aug 30 02:02:17 2022] DELETE /identity/v3/users/d01a8d9693e449f0a8567e352935e231 => generated 0 bytes in 821 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 02:02:18.858217 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:02:19.256637 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2084e430-3380-40fe-89b9-86f063165c0b None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:02:19.257574 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2084e430-3380-40fe-89b9-86f063165c0b None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:02:19.258049 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2084e430-3380-40fe-89b9-86f063165c0b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:02:19.320077 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-1cc875af-1de4-4354-9fbf-0431336cb001 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '43fc688221d54c3badbb8a6ba3b6d9e3'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:02:19.321810 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-1cc875af-1de4-4354-9fbf-0431336cb001 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '43fc688221d54c3badbb8a6ba3b6d9e3'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:02:19.376745 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-2084e430-3380-40fe-89b9-86f063165c0b None None] MFA Rules not processed for user `121c374d49a64335ab74a431cfbc1320`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:02:19.378472 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-2084e430-3380-40fe-89b9-86f063165c0b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:02:19.477326 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-1cc875af-1de4-4354-9fbf-0431336cb001 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '43fc688221d54c3badbb8a6ba3b6d9e3'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:02:19.478030 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-1cc875af-1de4-4354-9fbf-0431336cb001 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '43fc688221d54c3badbb8a6ba3b6d9e3'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:02:19.487543 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-1cc875af-1de4-4354-9fbf-0431336cb001 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '43fc688221d54c3badbb8a6ba3b6d9e3'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:02:19.514205 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 863/1739] 10.222.0.27 () {58 vars in 1224 bytes} [Tue Aug 30 02:02:18 2022] DELETE /identity/v3/users/43fc688221d54c3badbb8a6ba3b6d9e3 => generated 0 bytes in 838 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 30 02:02:19.518820 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71fdd73a-b1e0-46fa-a0c8-94404b2651f7 None None] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:02:19.519276 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71fdd73a-b1e0-46fa-a0c8-94404b2651f7 None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:02:19.519745 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71fdd73a-b1e0-46fa-a0c8-94404b2651f7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:02:19.550613 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.auth.core [None req-71fdd73a-b1e0-46fa-a0c8-94404b2651f7 None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81578) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:02:19.552657 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-71fdd73a-b1e0-46fa-a0c8-94404b2651f7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:02:19.580147 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 877/1740] 10.222.0.27 () {58 vars in 1009 bytes} [Tue Aug 30 02:02:19 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 325 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 02:02:19.687100 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 864/1741] 10.222.0.27 () {58 vars in 958 bytes} [Tue Aug 30 02:02:19 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 169 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 02:02:19.753127 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2084e430-3380-40fe-89b9-86f063165c0b None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:02:19.754732 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-2084e430-3380-40fe-89b9-86f063165c0b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:02:19.907810 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2084e430-3380-40fe-89b9-86f063165c0b None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:02:20.113887 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0aa7c35-d494-473f-948f-9c57bc14ba24 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f3ea1580c56f4814a92bbb2acf542905', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4562f3951fbe465280073e60a11e44e2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:02:20.114785 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0aa7c35-d494-473f-948f-9c57bc14ba24 service neutron] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:02:20.115268 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0aa7c35-d494-473f-948f-9c57bc14ba24 service neutron] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:02:20.115709 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0aa7c35-d494-473f-948f-9c57bc14ba24 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:02:20.160372 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0aa7c35-d494-473f-948f-9c57bc14ba24 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:02:20.167055 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0aa7c35-d494-473f-948f-9c57bc14ba24 service neutron] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:02:20.185792 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 878/1742] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Aug 30 02:02:19 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 433 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 02:02:20.244362 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71fdd73a-b1e0-46fa-a0c8-94404b2651f7 None None] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:02:20.246076 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-71fdd73a-b1e0-46fa-a0c8-94404b2651f7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:02:20.367170 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71fdd73a-b1e0-46fa-a0c8-94404b2651f7 None None] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:02:20.375838 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e1017bb-fa06-4159-b411-466012680ba9 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b4f0008fde424f22bc1f7307cf422774', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4562f3951fbe465280073e60a11e44e2', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:02:20.376703 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e1017bb-fa06-4159-b411-466012680ba9 service swift] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:02:20.376988 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e1017bb-fa06-4159-b411-466012680ba9 service swift] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:02:20.377277 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e1017bb-fa06-4159-b411-466012680ba9 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:02:20.471078 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e1017bb-fa06-4159-b411-466012680ba9 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:02:20.473852 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e1017bb-fa06-4159-b411-466012680ba9 service swift] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:02:20.484563 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0aa7c35-d494-473f-948f-9c57bc14ba24 service neutron] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:02:20.485267 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 865/1743] 10.222.0.27 () {60 vars in 1384 bytes} [Tue Aug 30 02:02:20 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 242 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Aug 30 02:02:20.488239 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-e0aa7c35-d494-473f-948f-9c57bc14ba24 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:02:20.517755 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0aa7c35-d494-473f-948f-9c57bc14ba24 service neutron] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:02:20.607320 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a032dfe-cc1c-4d42-8f0f-888484d05dea admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:02:20.612403 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a032dfe-cc1c-4d42-8f0f-888484d05dea admin admin] REQUEST_METHOD: `DELETE` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:02:20.613140 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a032dfe-cc1c-4d42-8f0f-888484d05dea admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:02:20.615755 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a032dfe-cc1c-4d42-8f0f-888484d05dea admin admin] PATH_INFO: `/v3/projects/b5e05350686f48a0a396f6f415734c9c` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:02:20.618881 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a032dfe-cc1c-4d42-8f0f-888484d05dea admin admin] RBAC: Authorizing `identity:delete_project(project_id=b5e05350686f48a0a396f6f415734c9c)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:02:20.628824 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a032dfe-cc1c-4d42-8f0f-888484d05dea admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:02:20.897216 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-8a032dfe-cc1c-4d42-8f0f-888484d05dea admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'b5e05350686f48a0a396f6f415734c9c', 'request_id': 'req-8a032dfe-cc1c-4d42-8f0f-888484d05dea'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:02:20.908185 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-8a032dfe-cc1c-4d42-8f0f-888484d05dea admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'b5e05350686f48a0a396f6f415734c9c', 'request_id': 'req-8a032dfe-cc1c-4d42-8f0f-888484d05dea'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:02:20.920005 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-8a032dfe-cc1c-4d42-8f0f-888484d05dea admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'b5e05350686f48a0a396f6f415734c9c', 'request_id': 'req-8a032dfe-cc1c-4d42-8f0f-888484d05dea'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:02:21.076756 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-8a032dfe-cc1c-4d42-8f0f-888484d05dea admin admin] The token cache is being invalidate because project b5e05350686f48a0a396f6f415734c9c was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81577) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 30 02:02:21.077872 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-8a032dfe-cc1c-4d42-8f0f-888484d05dea admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:02:21.080610 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-8a032dfe-cc1c-4d42-8f0f-888484d05dea admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:02:21.087342 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 879/1744] 10.222.0.27 () {58 vars in 1233 bytes} [Tue Aug 30 02:02:20 2022] DELETE /identity/v3/projects/b5e05350686f48a0a396f6f415734c9c => generated 0 bytes in 603 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 02:02:21.093128 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c612ac47-dcec-4d73-8cf3-209598509c75 None None] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:02:21.093412 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c612ac47-dcec-4d73-8cf3-209598509c75 None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:02:21.094104 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c612ac47-dcec-4d73-8cf3-209598509c75 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:02:21.181320 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.auth.core [None req-c612ac47-dcec-4d73-8cf3-209598509c75 None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81578) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:02:21.185291 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-c612ac47-dcec-4d73-8cf3-209598509c75 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:02:21.206746 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:02:21.356464 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 866/1745] 10.222.0.27 () {58 vars in 958 bytes} [Tue Aug 30 02:02:21 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 265 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 02:02:21.363394 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-110ccb81-d091-44ea-a16a-3f883ee47ebc None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:02:21.363941 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-110ccb81-d091-44ea-a16a-3f883ee47ebc None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:02:21.364937 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-110ccb81-d091-44ea-a16a-3f883ee47ebc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:02:21.461070 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-110ccb81-d091-44ea-a16a-3f883ee47ebc None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:02:21.469268 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-110ccb81-d091-44ea-a16a-3f883ee47ebc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:02:21.504778 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 880/1746] 10.222.0.27 () {58 vars in 958 bytes} [Tue Aug 30 02:02:21 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 142 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 02:02:21.505577 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c612ac47-dcec-4d73-8cf3-209598509c75 None None] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:02:21.517410 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c612ac47-dcec-4d73-8cf3-209598509c75 None None] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:02:21.550369 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f258aee-be1d-4df5-96e9-565f07a5ad91 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:02:21.551561 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f258aee-be1d-4df5-96e9-565f07a5ad91 admin admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:02:21.552543 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f258aee-be1d-4df5-96e9-565f07a5ad91 admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:02:21.554437 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f258aee-be1d-4df5-96e9-565f07a5ad91 admin admin] PATH_INFO: `/v3/domains` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:02:21.556035 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f258aee-be1d-4df5-96e9-565f07a5ad91 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81578) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 02:02:21.558578 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f258aee-be1d-4df5-96e9-565f07a5ad91 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:02:21.565035 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f258aee-be1d-4df5-96e9-565f07a5ad91 admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:02:21.573151 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-110ccb81-d091-44ea-a16a-3f883ee47ebc None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:02:21.581052 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-110ccb81-d091-44ea-a16a-3f883ee47ebc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:02:21.611013 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-110ccb81-d091-44ea-a16a-3f883ee47ebc None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:02:21.629994 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:02:21.643279 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 867/1747] 10.222.0.27 () {58 vars in 1153 bytes} [Tue Aug 30 02:02:21 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 02:02:21.643918 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58eb8c60-75b5-4d78-8a89-b50e613910f3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:02:21.644361 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f258aee-be1d-4df5-96e9-565f07a5ad91 admin admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:02:21.645559 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58eb8c60-75b5-4d78-8a89-b50e613910f3 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:02:21.652030 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58eb8c60-75b5-4d78-8a89-b50e613910f3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:02:21.652314 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58eb8c60-75b5-4d78-8a89-b50e613910f3 admin admin] PATH_INFO: `/v3/users/3e63fec0ec7649aa86422d775a018a49` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:02:21.658315 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f258aee-be1d-4df5-96e9-565f07a5ad91 admin admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:02:21.662130 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58eb8c60-75b5-4d78-8a89-b50e613910f3 admin admin] RBAC: Authorizing `identity:delete_user(user_id=3e63fec0ec7649aa86422d775a018a49)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:02:21.664107 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58eb8c60-75b5-4d78-8a89-b50e613910f3 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:02:21.664463 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5230a912-5a41-41b0-8b74-3f9801d9f40d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:02:21.665408 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5230a912-5a41-41b0-8b74-3f9801d9f40d admin admin] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:02:21.669308 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5230a912-5a41-41b0-8b74-3f9801d9f40d admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:02:21.669308 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5230a912-5a41-41b0-8b74-3f9801d9f40d admin admin] PATH_INFO: `/v3/projects` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:02:21.669308 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5230a912-5a41-41b0-8b74-3f9801d9f40d admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:02:21.671142 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5230a912-5a41-41b0-8b74-3f9801d9f40d admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:02:21.808220 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 868/1748] 10.222.0.27 () {60 vars in 1153 bytes} [Tue Aug 30 02:02:21 2022] POST /identity/v3/projects => generated 371 bytes in 164 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 02:02:21.819598 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5230a912-5a41-41b0-8b74-3f9801d9f40d admin admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:02:22.031180 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5230a912-5a41-41b0-8b74-3f9801d9f40d admin admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:02:22.031899 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-58eb8c60-75b5-4d78-8a89-b50e613910f3 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '3e63fec0ec7649aa86422d775a018a49'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:02:22.038632 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-58eb8c60-75b5-4d78-8a89-b50e613910f3 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '3e63fec0ec7649aa86422d775a018a49'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:02:22.040825 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-58eb8c60-75b5-4d78-8a89-b50e613910f3 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '3e63fec0ec7649aa86422d775a018a49'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:02:22.061040 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc58b148-129a-461d-9620-4ab8d2e7d3aa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:02:22.062580 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc58b148-129a-461d-9620-4ab8d2e7d3aa admin admin] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:02:22.062866 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc58b148-129a-461d-9620-4ab8d2e7d3aa admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:02:22.063146 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc58b148-129a-461d-9620-4ab8d2e7d3aa admin admin] PATH_INFO: `/v3/users` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:02:22.063800 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc58b148-129a-461d-9620-4ab8d2e7d3aa admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:02:22.070456 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc58b148-129a-461d-9620-4ab8d2e7d3aa admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:02:22.119934 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-58eb8c60-75b5-4d78-8a89-b50e613910f3 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '3e63fec0ec7649aa86422d775a018a49'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:02:22.189755 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-58eb8c60-75b5-4d78-8a89-b50e613910f3 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '3e63fec0ec7649aa86422d775a018a49'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:02:22.430956 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 869/1749] 10.222.0.27 () {60 vars in 1144 bytes} [Tue Aug 30 02:02:21 2022] POST /identity/v3/users => generated 286 bytes in 622 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 02:02:22.509154 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc58b148-129a-461d-9620-4ab8d2e7d3aa admin admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:02:22.520303 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-cc58b148-129a-461d-9620-4ab8d2e7d3aa admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:02:22.547290 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc58b148-129a-461d-9620-4ab8d2e7d3aa admin admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:02:22.564037 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7021defb-0dc5-4c30-9d9b-c20c7242a9ba admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:02:22.565363 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7021defb-0dc5-4c30-9d9b-c20c7242a9ba admin admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:02:22.565687 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7021defb-0dc5-4c30-9d9b-c20c7242a9ba admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:02:22.565968 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7021defb-0dc5-4c30-9d9b-c20c7242a9ba admin admin] PATH_INFO: `/v3/roles` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:02:22.566641 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7021defb-0dc5-4c30-9d9b-c20c7242a9ba admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:02:22.568276 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7021defb-0dc5-4c30-9d9b-c20c7242a9ba admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:02:22.600190 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 870/1750] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Aug 30 02:02:22 2022] GET /identity/v3/roles => generated 1420 bytes in 91 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 02:02:22.601624 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7021defb-0dc5-4c30-9d9b-c20c7242a9ba admin admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:02:22.646123 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7021defb-0dc5-4c30-9d9b-c20c7242a9ba admin admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:02:22.662305 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0ae9b80-d550-40f3-b6ec-e2d2ac4cfe63 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:02:22.663577 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0ae9b80-d550-40f3-b6ec-e2d2ac4cfe63 admin admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:02:22.664010 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0ae9b80-d550-40f3-b6ec-e2d2ac4cfe63 admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:02:22.664436 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0ae9b80-d550-40f3-b6ec-e2d2ac4cfe63 admin admin] PATH_INFO: `/v3/roles` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:02:22.665242 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0ae9b80-d550-40f3-b6ec-e2d2ac4cfe63 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:02:22.667047 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0ae9b80-d550-40f3-b6ec-e2d2ac4cfe63 admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:02:22.711132 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 871/1751] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Aug 30 02:02:22 2022] GET /identity/v3/roles => generated 1420 bytes in 110 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 02:02:22.717335 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0ae9b80-d550-40f3-b6ec-e2d2ac4cfe63 admin admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:02:22.753655 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 881/1752] 10.222.0.27 () {58 vars in 1224 bytes} [Tue Aug 30 02:02:21 2022] DELETE /identity/v3/users/3e63fec0ec7649aa86422d775a018a49 => generated 0 bytes in 1181 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 02:02:22.763203 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faa05e5e-8379-465b-b27a-4c28717f3247 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:02:22.764749 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faa05e5e-8379-465b-b27a-4c28717f3247 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:02:22.766200 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faa05e5e-8379-465b-b27a-4c28717f3247 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:02:22.791121 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0ae9b80-d550-40f3-b6ec-e2d2ac4cfe63 admin admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:02:22.859150 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-faa05e5e-8379-465b-b27a-4c28717f3247 None None] MFA Rules not processed for user `bda980735a1f4948ba695e6c2788f3b4`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:02:22.863308 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-faa05e5e-8379-465b-b27a-4c28717f3247 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:02:22.957296 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9858e2fe-dca8-4f03-9f27-ab7d24cc7b74 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:02:22.958056 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9858e2fe-dca8-4f03-9f27-ab7d24cc7b74 admin admin] REQUEST_METHOD: `PUT` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:02:22.958340 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9858e2fe-dca8-4f03-9f27-ab7d24cc7b74 admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:02:22.958613 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9858e2fe-dca8-4f03-9f27-ab7d24cc7b74 admin admin] PATH_INFO: `/v3/projects/d28fb6fcd2c641a6a70154e485d37a6b/users/99d693db122b49cd98cd55b0d21d5db2/roles/1fb81afcb9f245f3b989add94598a063` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:02:23.013497 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9858e2fe-dca8-4f03-9f27-ab7d24cc7b74 admin admin] RBAC: Authorizing `identity:create_grant(project_id=d28fb6fcd2c641a6a70154e485d37a6b, user_id=99d693db122b49cd98cd55b0d21d5db2, role_id=1fb81afcb9f245f3b989add94598a063)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:02:23.024105 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9858e2fe-dca8-4f03-9f27-ab7d24cc7b74 admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:02:23.126204 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 872/1753] 10.222.0.27 () {60 vars in 1483 bytes} [Tue Aug 30 02:02:22 2022] PUT /identity/v3/projects/d28fb6fcd2c641a6a70154e485d37a6b/users/99d693db122b49cd98cd55b0d21d5db2/roles/1fb81afcb9f245f3b989add94598a063 => generated 0 bytes in 410 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 02:02:23.130112 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88d70c12-a25c-4671-bc3d-4d57a57c8d57 None None] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:02:23.130112 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88d70c12-a25c-4671-bc3d-4d57a57c8d57 None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:02:23.130112 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88d70c12-a25c-4671-bc3d-4d57a57c8d57 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:02:23.142575 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 882/1754] 10.222.0.27 () {58 vars in 958 bytes} [Tue Aug 30 02:02:22 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 388 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 02:02:23.144649 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a8ec20d-de9d-4b3c-ac77-3dff7a0fdc06 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:02:23.147559 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a8ec20d-de9d-4b3c-ac77-3dff7a0fdc06 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:02:23.148045 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a8ec20d-de9d-4b3c-ac77-3dff7a0fdc06 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:02:23.204027 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.auth.core [None req-88d70c12-a25c-4671-bc3d-4d57a57c8d57 None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81578) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:02:23.213835 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-88d70c12-a25c-4671-bc3d-4d57a57c8d57 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:02:23.283931 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-2a8ec20d-de9d-4b3c-ac77-3dff7a0fdc06 None None] MFA Rules not processed for user `99d693db122b49cd98cd55b0d21d5db2`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:02:23.285251 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-2a8ec20d-de9d-4b3c-ac77-3dff7a0fdc06 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:02:23.440320 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 883/1755] 10.222.0.27 () {58 vars in 958 bytes} [Tue Aug 30 02:02:23 2022] POST /identity/v3/auth/tokens => generated 3248 bytes in 297 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 02:02:23.441943 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ebb9bbb-71ee-4dfd-9f35-15da759d6239 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:02:23.450740 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ebb9bbb-71ee-4dfd-9f35-15da759d6239 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:02:23.450740 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ebb9bbb-71ee-4dfd-9f35-15da759d6239 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:02:23.475882 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-5ebb9bbb-71ee-4dfd-9f35-15da759d6239 None None] MFA Rules not processed for user `f3ea1580c56f4814a92bbb2acf542905`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:02:23.477636 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-5ebb9bbb-71ee-4dfd-9f35-15da759d6239 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:02:23.516055 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 873/1756] 10.222.0.27 () {58 vars in 958 bytes} [Tue Aug 30 02:02:23 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 389 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 02:02:23.516930 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88d70c12-a25c-4671-bc3d-4d57a57c8d57 None None] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:02:23.531244 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-88d70c12-a25c-4671-bc3d-4d57a57c8d57 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:02:23.559996 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 884/1757] 10.222.0.42 () {58 vars in 1015 bytes} [Tue Aug 30 02:02:23 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 119 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 02:02:23.596295 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:02:23.609850 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ebb9bbb-71ee-4dfd-9f35-15da759d6239 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:02:23.611288 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-5ebb9bbb-71ee-4dfd-9f35-15da759d6239 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:02:23.626837 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ebb9bbb-71ee-4dfd-9f35-15da759d6239 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:02:23.634050 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13b92f1c-3dd6-485d-9167-850eef205b9b service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f3ea1580c56f4814a92bbb2acf542905', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4562f3951fbe465280073e60a11e44e2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:02:23.636372 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13b92f1c-3dd6-485d-9167-850eef205b9b service neutron] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:02:23.647244 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13b92f1c-3dd6-485d-9167-850eef205b9b service neutron] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:02:23.649348 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13b92f1c-3dd6-485d-9167-850eef205b9b service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:02:23.672408 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13b92f1c-3dd6-485d-9167-850eef205b9b service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:02:23.677136 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13b92f1c-3dd6-485d-9167-850eef205b9b service neutron] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:02:23.739723 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 885/1758] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Aug 30 02:02:23 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 130 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 30 02:02:23.740605 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13b92f1c-3dd6-485d-9167-850eef205b9b service neutron] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:02:23.767125 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88d70c12-a25c-4671-bc3d-4d57a57c8d57 None None] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:02:23.771252 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13b92f1c-3dd6-485d-9167-850eef205b9b service neutron] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:02:23.790308 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3aaf9cac-bc62-4c1d-a3cc-c7fe80e99cba service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ab936b3767a8417aa80c8c8f1b84b667', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4562f3951fbe465280073e60a11e44e2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:02:23.790983 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3aaf9cac-bc62-4c1d-a3cc-c7fe80e99cba service cinder] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:02:23.791267 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3aaf9cac-bc62-4c1d-a3cc-c7fe80e99cba service cinder] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:02:23.791539 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3aaf9cac-bc62-4c1d-a3cc-c7fe80e99cba service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:02:23.800497 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-caa7108e-a337-4ef2-a7a0-d7f20b8796b0 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f3ea1580c56f4814a92bbb2acf542905', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4562f3951fbe465280073e60a11e44e2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:02:23.801487 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-caa7108e-a337-4ef2-a7a0-d7f20b8796b0 service neutron] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:02:23.808017 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-caa7108e-a337-4ef2-a7a0-d7f20b8796b0 service neutron] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:02:23.808570 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-caa7108e-a337-4ef2-a7a0-d7f20b8796b0 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:02:23.819575 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-caa7108e-a337-4ef2-a7a0-d7f20b8796b0 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:02:23.829901 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-caa7108e-a337-4ef2-a7a0-d7f20b8796b0 service neutron] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:02:23.834724 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3aaf9cac-bc62-4c1d-a3cc-c7fe80e99cba service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:02:23.836161 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3aaf9cac-bc62-4c1d-a3cc-c7fe80e99cba service cinder] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:02:23.839496 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 886/1759] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Aug 30 02:02:23 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 99 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 02:02:23.840580 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-caa7108e-a337-4ef2-a7a0-d7f20b8796b0 service neutron] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:02:23.844810 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-caa7108e-a337-4ef2-a7a0-d7f20b8796b0 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:02:23.886783 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 874/1760] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Aug 30 02:02:23 2022] GET /identity/v3/auth/tokens => generated 3248 bytes in 370 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 02:02:23.887841 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3aaf9cac-bc62-4c1d-a3cc-c7fe80e99cba service cinder] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:02:23.894508 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-caa7108e-a337-4ef2-a7a0-d7f20b8796b0 service neutron] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:02:23.895601 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-3aaf9cac-bc62-4c1d-a3cc-c7fe80e99cba service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:02:23.910951 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48b89425-24bb-4e21-891d-a5bf77f4fbbb service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f3ea1580c56f4814a92bbb2acf542905', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4562f3951fbe465280073e60a11e44e2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:02:23.911982 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48b89425-24bb-4e21-891d-a5bf77f4fbbb service neutron] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:02:23.912549 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48b89425-24bb-4e21-891d-a5bf77f4fbbb service neutron] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:02:23.913152 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48b89425-24bb-4e21-891d-a5bf77f4fbbb service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:02:23.941330 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48b89425-24bb-4e21-891d-a5bf77f4fbbb service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:02:23.941892 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3aaf9cac-bc62-4c1d-a3cc-c7fe80e99cba service cinder] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:02:23.953976 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48b89425-24bb-4e21-891d-a5bf77f4fbbb service neutron] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:02:23.955862 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47795d24-9ece-4e10-a91c-6e6c314989ab admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:02:23.956804 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47795d24-9ece-4e10-a91c-6e6c314989ab admin admin] REQUEST_METHOD: `DELETE` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:02:23.957126 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47795d24-9ece-4e10-a91c-6e6c314989ab admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:02:23.957400 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47795d24-9ece-4e10-a91c-6e6c314989ab admin admin] PATH_INFO: `/v3/projects/1b77ab6a0dc743f993aa7cfe813a88ba` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:02:23.958995 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47795d24-9ece-4e10-a91c-6e6c314989ab admin admin] RBAC: Authorizing `identity:delete_project(project_id=1b77ab6a0dc743f993aa7cfe813a88ba)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:02:23.964049 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47795d24-9ece-4e10-a91c-6e6c314989ab admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:02:23.988923 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 887/1761] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Aug 30 02:02:23 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 148 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 02:02:23.995914 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48b89425-24bb-4e21-891d-a5bf77f4fbbb service neutron] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:02:24.040916 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:02:24.047657 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48b89425-24bb-4e21-891d-a5bf77f4fbbb service neutron] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:02:24.064477 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12cccda2-40f3-4f0b-8bf5-f1ce7e4a108e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:02:24.076490 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12cccda2-40f3-4f0b-8bf5-f1ce7e4a108e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:02:24.076490 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12cccda2-40f3-4f0b-8bf5-f1ce7e4a108e admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:02:24.076490 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12cccda2-40f3-4f0b-8bf5-f1ce7e4a108e admin admin] PATH_INFO: `/v3/users/99d693db122b49cd98cd55b0d21d5db2` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:02:24.080000 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12cccda2-40f3-4f0b-8bf5-f1ce7e4a108e admin admin] RBAC: Authorizing `identity:delete_user(user_id=99d693db122b49cd98cd55b0d21d5db2)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:02:24.090340 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12cccda2-40f3-4f0b-8bf5-f1ce7e4a108e admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:02:24.122790 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-47795d24-9ece-4e10-a91c-6e6c314989ab admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '1b77ab6a0dc743f993aa7cfe813a88ba', 'request_id': 'req-47795d24-9ece-4e10-a91c-6e6c314989ab'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:02:24.144348 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-47795d24-9ece-4e10-a91c-6e6c314989ab admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '1b77ab6a0dc743f993aa7cfe813a88ba', 'request_id': 'req-47795d24-9ece-4e10-a91c-6e6c314989ab'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:02:24.161139 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-47795d24-9ece-4e10-a91c-6e6c314989ab admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '1b77ab6a0dc743f993aa7cfe813a88ba', 'request_id': 'req-47795d24-9ece-4e10-a91c-6e6c314989ab'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:02:24.269940 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:02:24.281948 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-47795d24-9ece-4e10-a91c-6e6c314989ab admin admin] The token cache is being invalidate because project 1b77ab6a0dc743f993aa7cfe813a88ba was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81578) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 30 02:02:24.282387 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-47795d24-9ece-4e10-a91c-6e6c314989ab admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:02:24.283856 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-47795d24-9ece-4e10-a91c-6e6c314989ab admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:02:24.291945 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 875/1762] 10.222.0.27 () {58 vars in 1233 bytes} [Tue Aug 30 02:02:23 2022] DELETE /identity/v3/projects/1b77ab6a0dc743f993aa7cfe813a88ba => generated 0 bytes in 402 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 02:02:24.294428 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47795d24-9ece-4e10-a91c-6e6c314989ab admin admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:02:24.298170 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-47795d24-9ece-4e10-a91c-6e6c314989ab admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:02:24.419708 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-12cccda2-40f3-4f0b-8bf5-f1ce7e4a108e admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '99d693db122b49cd98cd55b0d21d5db2'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:02:24.425892 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-12cccda2-40f3-4f0b-8bf5-f1ce7e4a108e admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '99d693db122b49cd98cd55b0d21d5db2'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:02:24.427356 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-12cccda2-40f3-4f0b-8bf5-f1ce7e4a108e admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '99d693db122b49cd98cd55b0d21d5db2'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:02:24.474523 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-12cccda2-40f3-4f0b-8bf5-f1ce7e4a108e admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '99d693db122b49cd98cd55b0d21d5db2'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:02:24.502296 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-12cccda2-40f3-4f0b-8bf5-f1ce7e4a108e admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '99d693db122b49cd98cd55b0d21d5db2'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:02:24.546171 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47795d24-9ece-4e10-a91c-6e6c314989ab admin admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:02:24.557628 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7003ae1d-43bf-4423-9769-65fc7458e414 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:02:24.557628 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7003ae1d-43bf-4423-9769-65fc7458e414 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:02:24.557628 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7003ae1d-43bf-4423-9769-65fc7458e414 admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:02:24.557628 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7003ae1d-43bf-4423-9769-65fc7458e414 admin admin] PATH_INFO: `/v3/users/e8898371204546a89e21b6e34aaa04eb` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:02:24.558458 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7003ae1d-43bf-4423-9769-65fc7458e414 admin admin] RBAC: Authorizing `identity:delete_user(user_id=e8898371204546a89e21b6e34aaa04eb)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:02:24.562050 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7003ae1d-43bf-4423-9769-65fc7458e414 admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:02:24.588220 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 888/1763] 10.222.0.27 () {58 vars in 1224 bytes} [Tue Aug 30 02:02:23 2022] DELETE /identity/v3/users/99d693db122b49cd98cd55b0d21d5db2 => generated 0 bytes in 599 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 02:02:24.593302 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd551006-5177-45fa-8b32-e05b6c125e9c None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:02:24.594197 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd551006-5177-45fa-8b32-e05b6c125e9c None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:02:24.594670 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd551006-5177-45fa-8b32-e05b6c125e9c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:02:24.644439 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-bd551006-5177-45fa-8b32-e05b6c125e9c None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:02:24.645811 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-bd551006-5177-45fa-8b32-e05b6c125e9c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:02:24.772803 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=87,keystone:DELETE=12,keystone:INSERT=1 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:02:24.823975 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 889/1764] 10.222.0.27 () {58 vars in 958 bytes} [Tue Aug 30 02:02:24 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 232 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 02:02:24.839710 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-7003ae1d-43bf-4423-9769-65fc7458e414 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'e8898371204546a89e21b6e34aaa04eb'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:02:24.846060 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-7003ae1d-43bf-4423-9769-65fc7458e414 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'e8898371204546a89e21b6e34aaa04eb'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:02:24.918858 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd551006-5177-45fa-8b32-e05b6c125e9c None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:02:24.920969 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-bd551006-5177-45fa-8b32-e05b6c125e9c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:02:24.973941 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-7003ae1d-43bf-4423-9769-65fc7458e414 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'e8898371204546a89e21b6e34aaa04eb'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:02:24.974631 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-7003ae1d-43bf-4423-9769-65fc7458e414 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'e8898371204546a89e21b6e34aaa04eb'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:02:24.996653 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-7003ae1d-43bf-4423-9769-65fc7458e414 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'e8898371204546a89e21b6e34aaa04eb'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:02:25.069986 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 876/1765] 10.222.0.27 () {58 vars in 1224 bytes} [Tue Aug 30 02:02:24 2022] DELETE /identity/v3/users/e8898371204546a89e21b6e34aaa04eb => generated 0 bytes in 774 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 02:02:25.082966 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd551006-5177-45fa-8b32-e05b6c125e9c None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:02:25.269896 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44ab44b9-6500-4ba3-8768-ad72040c657a service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f3ea1580c56f4814a92bbb2acf542905', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4562f3951fbe465280073e60a11e44e2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:02:25.274442 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44ab44b9-6500-4ba3-8768-ad72040c657a service neutron] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:02:25.274970 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44ab44b9-6500-4ba3-8768-ad72040c657a service neutron] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:02:25.275577 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44ab44b9-6500-4ba3-8768-ad72040c657a service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:02:25.286423 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-44ab44b9-6500-4ba3-8768-ad72040c657a service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:02:25.298760 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc09ebd8-bfef-4882-bf92-7d3f487f12f1 None None] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:02:25.299567 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc09ebd8-bfef-4882-bf92-7d3f487f12f1 None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:02:25.300198 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc09ebd8-bfef-4882-bf92-7d3f487f12f1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:02:25.383567 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.auth.core [None req-cc09ebd8-bfef-4882-bf92-7d3f487f12f1 None None] MFA Rules not processed for user `7af1c1123b2e458b84248865287bd00e`. Rule list: `[]` (Enabled: `True`). {{(pid=81578) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:02:25.385348 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-cc09ebd8-bfef-4882-bf92-7d3f487f12f1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:02:25.566122 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 877/1766] 10.222.0.27 () {58 vars in 958 bytes} [Tue Aug 30 02:02:25 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 264 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 02:02:25.574623 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44ab44b9-6500-4ba3-8768-ad72040c657a service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:02:25.580597 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44ab44b9-6500-4ba3-8768-ad72040c657a service neutron] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:02:25.619948 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 890/1767] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Aug 30 02:02:24 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 696 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 02:02:26.461059 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc09ebd8-bfef-4882-bf92-7d3f487f12f1 None None] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:02:26.462704 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-cc09ebd8-bfef-4882-bf92-7d3f487f12f1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:02:26.503008 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc09ebd8-bfef-4882-bf92-7d3f487f12f1 None None] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:02:26.527007 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20b9ee0d-6ac1-4781-8928-bb3e863dc781 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:02:26.529893 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20b9ee0d-6ac1-4781-8928-bb3e863dc781 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:02:26.529893 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20b9ee0d-6ac1-4781-8928-bb3e863dc781 admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:02:26.529893 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20b9ee0d-6ac1-4781-8928-bb3e863dc781 admin admin] PATH_INFO: `/v3/projects/0da6a09895c54fe3a759d06eb7773159` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:02:26.531321 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20b9ee0d-6ac1-4781-8928-bb3e863dc781 admin admin] RBAC: Authorizing `identity:delete_project(project_id=0da6a09895c54fe3a759d06eb7773159)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:02:26.533476 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20b9ee0d-6ac1-4781-8928-bb3e863dc781 admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:02:26.799049 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-20b9ee0d-6ac1-4781-8928-bb3e863dc781 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '0da6a09895c54fe3a759d06eb7773159', 'request_id': 'req-20b9ee0d-6ac1-4781-8928-bb3e863dc781'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:02:26.814229 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-20b9ee0d-6ac1-4781-8928-bb3e863dc781 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '0da6a09895c54fe3a759d06eb7773159', 'request_id': 'req-20b9ee0d-6ac1-4781-8928-bb3e863dc781'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:02:26.835491 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-20b9ee0d-6ac1-4781-8928-bb3e863dc781 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '0da6a09895c54fe3a759d06eb7773159', 'request_id': 'req-20b9ee0d-6ac1-4781-8928-bb3e863dc781'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:02:26.995382 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-20b9ee0d-6ac1-4781-8928-bb3e863dc781 admin admin] The token cache is being invalidate because project 0da6a09895c54fe3a759d06eb7773159 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81578) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 30 02:02:26.995382 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-20b9ee0d-6ac1-4781-8928-bb3e863dc781 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:02:26.995382 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-20b9ee0d-6ac1-4781-8928-bb3e863dc781 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:02:27.002548 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 878/1768] 10.222.0.27 () {58 vars in 1233 bytes} [Tue Aug 30 02:02:26 2022] DELETE /identity/v3/projects/0da6a09895c54fe3a759d06eb7773159 => generated 0 bytes in 539 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 02:02:27.003997 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa9d5bbf-647a-4dc7-89da-5f55d51c2c4a None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:02:27.004511 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa9d5bbf-647a-4dc7-89da-5f55d51c2c4a None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:02:27.004636 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa9d5bbf-647a-4dc7-89da-5f55d51c2c4a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:02:27.030988 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-aa9d5bbf-647a-4dc7-89da-5f55d51c2c4a None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:02:27.033291 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-aa9d5bbf-647a-4dc7-89da-5f55d51c2c4a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:02:27.210415 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 891/1769] 10.222.0.27 () {58 vars in 958 bytes} [Tue Aug 30 02:02:27 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 207 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 02:02:27.211522 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a297c41-cf2d-491a-ba20-8a1db6e9c361 None None] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:02:27.212029 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a297c41-cf2d-491a-ba20-8a1db6e9c361 None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:02:27.212401 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a297c41-cf2d-491a-ba20-8a1db6e9c361 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:02:27.275592 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.auth.core [None req-6a297c41-cf2d-491a-ba20-8a1db6e9c361 None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81578) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:02:27.278411 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-6a297c41-cf2d-491a-ba20-8a1db6e9c361 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:02:27.323104 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 879/1770] 10.222.0.27 () {58 vars in 958 bytes} [Tue Aug 30 02:02:27 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 02:02:27.324983 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa9d5bbf-647a-4dc7-89da-5f55d51c2c4a None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:02:27.346347 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa9d5bbf-647a-4dc7-89da-5f55d51c2c4a None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:02:27.403459 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c7c5be9-73ec-49cb-8a3b-173e85b610a2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:02:27.426153 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c7c5be9-73ec-49cb-8a3b-173e85b610a2 admin admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:02:27.429038 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c7c5be9-73ec-49cb-8a3b-173e85b610a2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:02:27.430745 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c7c5be9-73ec-49cb-8a3b-173e85b610a2 admin admin] PATH_INFO: `/v3/domains` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:02:27.432213 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c7c5be9-73ec-49cb-8a3b-173e85b610a2 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 02:02:27.432552 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c7c5be9-73ec-49cb-8a3b-173e85b610a2 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:02:27.434215 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c7c5be9-73ec-49cb-8a3b-173e85b610a2 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:02:27.484097 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 892/1771] 10.222.0.27 () {58 vars in 1153 bytes} [Tue Aug 30 02:02:27 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 02:02:27.487173 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a297c41-cf2d-491a-ba20-8a1db6e9c361 None None] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:02:27.505179 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a297c41-cf2d-491a-ba20-8a1db6e9c361 None None] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:02:27.515659 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:02:27.554651 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9c8b744-cd10-4189-a31b-eb874f7488d5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:02:27.555402 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9c8b744-cd10-4189-a31b-eb874f7488d5 admin admin] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:02:27.555685 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9c8b744-cd10-4189-a31b-eb874f7488d5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:02:27.555964 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9c8b744-cd10-4189-a31b-eb874f7488d5 admin admin] PATH_INFO: `/v3/projects` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:02:27.556611 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9c8b744-cd10-4189-a31b-eb874f7488d5 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:02:27.561181 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9c8b744-cd10-4189-a31b-eb874f7488d5 admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:02:27.766924 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 880/1772] 10.222.0.27 () {60 vars in 1153 bytes} [Tue Aug 30 02:02:27 2022] POST /identity/v3/projects => generated 373 bytes in 280 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 02:02:27.771521 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c7c5be9-73ec-49cb-8a3b-173e85b610a2 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:02:27.785129 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c7c5be9-73ec-49cb-8a3b-173e85b610a2 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:02:27.926859 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c18e23f1-a92a-4075-b15d-9e1ea82996eb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:02:27.940406 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c18e23f1-a92a-4075-b15d-9e1ea82996eb admin admin] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:02:27.943433 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c18e23f1-a92a-4075-b15d-9e1ea82996eb admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:02:27.944652 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c18e23f1-a92a-4075-b15d-9e1ea82996eb admin admin] PATH_INFO: `/v3/users` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:02:27.946499 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c18e23f1-a92a-4075-b15d-9e1ea82996eb admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:02:27.952122 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c18e23f1-a92a-4075-b15d-9e1ea82996eb admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:02:28.131188 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 893/1773] 10.222.0.27 () {60 vars in 1144 bytes} [Tue Aug 30 02:02:27 2022] POST /identity/v3/users => generated 287 bytes in 357 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 02:02:28.132434 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9c8b744-cd10-4189-a31b-eb874f7488d5 admin admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:02:28.150705 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9c8b744-cd10-4189-a31b-eb874f7488d5 admin admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:02:28.158903 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b90b2549-33f9-4254-a710-0ea405f8694d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:02:28.159522 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b90b2549-33f9-4254-a710-0ea405f8694d admin admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:02:28.159963 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b90b2549-33f9-4254-a710-0ea405f8694d admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:02:28.160377 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b90b2549-33f9-4254-a710-0ea405f8694d admin admin] PATH_INFO: `/v3/roles` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:02:28.161242 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b90b2549-33f9-4254-a710-0ea405f8694d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:02:28.163074 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b90b2549-33f9-4254-a710-0ea405f8694d admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:02:28.188681 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 881/1774] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Aug 30 02:02:28 2022] GET /identity/v3/roles => generated 1420 bytes in 57 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 02:02:28.192866 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c18e23f1-a92a-4075-b15d-9e1ea82996eb admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:02:28.208820 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c18e23f1-a92a-4075-b15d-9e1ea82996eb admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:02:28.220867 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29c5c8e8-548e-43e9-aac7-a96a0db7d9a4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:02:28.221523 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29c5c8e8-548e-43e9-aac7-a96a0db7d9a4 admin admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:02:28.221806 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29c5c8e8-548e-43e9-aac7-a96a0db7d9a4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:02:28.222119 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29c5c8e8-548e-43e9-aac7-a96a0db7d9a4 admin admin] PATH_INFO: `/v3/roles` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:02:28.222725 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29c5c8e8-548e-43e9-aac7-a96a0db7d9a4 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:02:28.224381 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29c5c8e8-548e-43e9-aac7-a96a0db7d9a4 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:02:28.235624 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07ed2129-794f-466a-8da7-117a14da0245 None None] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:02:28.235907 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07ed2129-794f-466a-8da7-117a14da0245 None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:02:28.236381 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07ed2129-794f-466a-8da7-117a14da0245 None None] PATH_INFO: `/` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:02:28.237976 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 882/1775] 10.222.0.27 () {54 vars in 921 bytes} [Tue Aug 30 02:02:28 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 02:02:28.254566 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc890191-18c0-4130-a207-f27c1da0c3ca None None] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:02:28.255093 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc890191-18c0-4130-a207-f27c1da0c3ca None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:02:28.255568 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc890191-18c0-4130-a207-f27c1da0c3ca None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:02:28.271782 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 894/1776] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Aug 30 02:02:28 2022] GET /identity/v3/roles => generated 1420 bytes in 79 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 02:02:28.272724 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29c5c8e8-548e-43e9-aac7-a96a0db7d9a4 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:02:28.289405 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-29c5c8e8-548e-43e9-aac7-a96a0db7d9a4 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:02:28.298693 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.auth.core [None req-bc890191-18c0-4130-a207-f27c1da0c3ca None None] MFA Rules not processed for user `d64f2e71361644589bec8a06e6eceed1`. Rule list: `[]` (Enabled: `True`). {{(pid=81578) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:02:28.300104 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-bc890191-18c0-4130-a207-f27c1da0c3ca None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:02:28.315301 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=94,keystone:DELETE=5,keystone:INSERT=1 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:02:28.317359 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29c5c8e8-548e-43e9-aac7-a96a0db7d9a4 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:02:28.378262 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-accc0d55-28c6-4cde-b4cb-397f3f3cf888 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:02:28.382145 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-accc0d55-28c6-4cde-b4cb-397f3f3cf888 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:02:28.382589 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-accc0d55-28c6-4cde-b4cb-397f3f3cf888 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:02:28.383121 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-accc0d55-28c6-4cde-b4cb-397f3f3cf888 admin admin] PATH_INFO: `/v3/projects/d28fb6fcd2c641a6a70154e485d37a6b` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:02:28.404052 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-accc0d55-28c6-4cde-b4cb-397f3f3cf888 admin admin] RBAC: Authorizing `identity:delete_project(project_id=d28fb6fcd2c641a6a70154e485d37a6b)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:02:28.408054 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-accc0d55-28c6-4cde-b4cb-397f3f3cf888 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:02:28.471410 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 883/1777] 10.222.0.27 () {58 vars in 1018 bytes} [Tue Aug 30 02:02:28 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 218 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 02:02:28.472349 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc890191-18c0-4130-a207-f27c1da0c3ca None None] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:02:28.491660 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc890191-18c0-4130-a207-f27c1da0c3ca None None] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:02:28.502952 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fefdd2e-03ef-4b3b-955a-12ac5f6894ea admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:02:28.504203 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fefdd2e-03ef-4b3b-955a-12ac5f6894ea admin admin] REQUEST_METHOD: `PUT` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:02:28.504672 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fefdd2e-03ef-4b3b-955a-12ac5f6894ea admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:02:28.505047 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fefdd2e-03ef-4b3b-955a-12ac5f6894ea admin admin] PATH_INFO: `/v3/projects/e0d38af4bc984e459007eccc60f3e117/users/4194355610df4d95bbcbd46b25426c63/roles/1fb81afcb9f245f3b989add94598a063` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:02:28.547049 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fefdd2e-03ef-4b3b-955a-12ac5f6894ea admin admin] RBAC: Authorizing `identity:create_grant(project_id=e0d38af4bc984e459007eccc60f3e117, user_id=4194355610df4d95bbcbd46b25426c63, role_id=1fb81afcb9f245f3b989add94598a063)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:02:28.550117 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fefdd2e-03ef-4b3b-955a-12ac5f6894ea admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:02:28.726694 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 884/1778] 10.222.0.27 () {60 vars in 1483 bytes} [Tue Aug 30 02:02:28 2022] PUT /identity/v3/projects/e0d38af4bc984e459007eccc60f3e117/users/4194355610df4d95bbcbd46b25426c63/roles/1fb81afcb9f245f3b989add94598a063 => generated 0 bytes in 254 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 02:02:28.727944 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fefdd2e-03ef-4b3b-955a-12ac5f6894ea admin admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:02:28.737673 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-7fefdd2e-03ef-4b3b-955a-12ac5f6894ea admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:02:28.762770 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-accc0d55-28c6-4cde-b4cb-397f3f3cf888 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'd28fb6fcd2c641a6a70154e485d37a6b', 'request_id': 'req-accc0d55-28c6-4cde-b4cb-397f3f3cf888'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:02:28.770773 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-accc0d55-28c6-4cde-b4cb-397f3f3cf888 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'd28fb6fcd2c641a6a70154e485d37a6b', 'request_id': 'req-accc0d55-28c6-4cde-b4cb-397f3f3cf888'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:02:28.790554 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-accc0d55-28c6-4cde-b4cb-397f3f3cf888 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'd28fb6fcd2c641a6a70154e485d37a6b', 'request_id': 'req-accc0d55-28c6-4cde-b4cb-397f3f3cf888'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:02:28.836060 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:02:28.916070 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-accc0d55-28c6-4cde-b4cb-397f3f3cf888 admin admin] The token cache is being invalidate because project d28fb6fcd2c641a6a70154e485d37a6b was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81577) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 30 02:02:28.916362 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-accc0d55-28c6-4cde-b4cb-397f3f3cf888 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:02:28.917974 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-accc0d55-28c6-4cde-b4cb-397f3f3cf888 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:02:28.928885 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 895/1779] 10.222.0.27 () {58 vars in 1233 bytes} [Tue Aug 30 02:02:28 2022] DELETE /identity/v3/projects/d28fb6fcd2c641a6a70154e485d37a6b => generated 0 bytes in 657 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 02:02:28.930462 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1e119bf-2e1a-4bd9-8c5c-857105ed4817 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:02:28.930890 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1e119bf-2e1a-4bd9-8c5c-857105ed4817 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:02:28.931295 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1e119bf-2e1a-4bd9-8c5c-857105ed4817 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:02:29.009934 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-d1e119bf-2e1a-4bd9-8c5c-857105ed4817 None None] MFA Rules not processed for user `4194355610df4d95bbcbd46b25426c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:02:29.012772 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-d1e119bf-2e1a-4bd9-8c5c-857105ed4817 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:02:29.065019 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fefdd2e-03ef-4b3b-955a-12ac5f6894ea admin admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:02:29.187170 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 896/1780] 10.222.0.27 () {58 vars in 958 bytes} [Tue Aug 30 02:02:28 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 257 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 02:02:29.200076 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-068b143f-9a83-435f-b6af-37fe17d15e80 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:02:29.200076 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-068b143f-9a83-435f-b6af-37fe17d15e80 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:02:29.200076 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-068b143f-9a83-435f-b6af-37fe17d15e80 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:02:29.249121 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-068b143f-9a83-435f-b6af-37fe17d15e80 None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:02:29.250520 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-068b143f-9a83-435f-b6af-37fe17d15e80 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:02:29.359914 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a88ee734-dac0-4f73-85f8-e4fec56f459a service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd64f2e71361644589bec8a06e6eceed1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4562f3951fbe465280073e60a11e44e2', 'project_domain_id': 'default', 'roles': ['member', 'service', 'admin', 'ResellerAdmin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:02:29.360740 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a88ee734-dac0-4f73-85f8-e4fec56f459a service nova] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:02:29.361022 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a88ee734-dac0-4f73-85f8-e4fec56f459a service nova] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:02:29.361293 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a88ee734-dac0-4f73-85f8-e4fec56f459a service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:02:29.363545 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 897/1781] 10.222.0.27 () {58 vars in 958 bytes} [Tue Aug 30 02:02:29 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 176 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 02:02:29.365385 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-596f5dc2-ae02-48ff-b9e4-c4a27dc9beb8 None None] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:02:29.365838 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-596f5dc2-ae02-48ff-b9e4-c4a27dc9beb8 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:02:29.366709 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-596f5dc2-ae02-48ff-b9e4-c4a27dc9beb8 None None] PATH_INFO: `/` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:02:29.367807 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-a88ee734-dac0-4f73-85f8-e4fec56f459a service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:02:29.370810 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 898/1782] 10.222.0.27 () {54 vars in 921 bytes} [Tue Aug 30 02:02:29 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 02:02:29.372364 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-596f5dc2-ae02-48ff-b9e4-c4a27dc9beb8 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:02:29.381442 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-596f5dc2-ae02-48ff-b9e4-c4a27dc9beb8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:02:29.403764 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a88ee734-dac0-4f73-85f8-e4fec56f459a service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:02:29.410907 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-596f5dc2-ae02-48ff-b9e4-c4a27dc9beb8 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:02:29.413266 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a88ee734-dac0-4f73-85f8-e4fec56f459a service nova] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:02:29.423003 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4485d043-5e4a-4d87-8514-4ec70c86e3e4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:02:29.423659 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4485d043-5e4a-4d87-8514-4ec70c86e3e4 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:02:29.424057 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4485d043-5e4a-4d87-8514-4ec70c86e3e4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:02:29.424329 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4485d043-5e4a-4d87-8514-4ec70c86e3e4 admin admin] PATH_INFO: `/v3/projects/34e6a886fdef4899a6261200d9ecd640` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:02:29.430401 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4485d043-5e4a-4d87-8514-4ec70c86e3e4 admin admin] RBAC: Authorizing `identity:delete_project(project_id=34e6a886fdef4899a6261200d9ecd640)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:02:29.433121 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4485d043-5e4a-4d87-8514-4ec70c86e3e4 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:02:29.448598 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 885/1783] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Aug 30 02:02:28 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 721 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 02:02:29.449735 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a88ee734-dac0-4f73-85f8-e4fec56f459a service nova] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:02:29.452082 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-a88ee734-dac0-4f73-85f8-e4fec56f459a service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:02:29.496257 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a88ee734-dac0-4f73-85f8-e4fec56f459a service nova] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:02:29.519002 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49dce7cf-414f-4ba2-8ea9-e46aa300d8f0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:02:29.520701 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49dce7cf-414f-4ba2-8ea9-e46aa300d8f0 admin admin] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:02:29.522967 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49dce7cf-414f-4ba2-8ea9-e46aa300d8f0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:02:29.527300 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49dce7cf-414f-4ba2-8ea9-e46aa300d8f0 admin admin] PATH_INFO: `/v3/projects` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:02:29.527300 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49dce7cf-414f-4ba2-8ea9-e46aa300d8f0 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:02:29.536268 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49dce7cf-414f-4ba2-8ea9-e46aa300d8f0 admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:02:29.692532 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-4485d043-5e4a-4d87-8514-4ec70c86e3e4 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '34e6a886fdef4899a6261200d9ecd640', 'request_id': 'req-4485d043-5e4a-4d87-8514-4ec70c86e3e4'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:02:29.706615 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-4485d043-5e4a-4d87-8514-4ec70c86e3e4 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '34e6a886fdef4899a6261200d9ecd640', 'request_id': 'req-4485d043-5e4a-4d87-8514-4ec70c86e3e4'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:02:29.734054 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-4485d043-5e4a-4d87-8514-4ec70c86e3e4 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '34e6a886fdef4899a6261200d9ecd640', 'request_id': 'req-4485d043-5e4a-4d87-8514-4ec70c86e3e4'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:02:29.881493 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-4485d043-5e4a-4d87-8514-4ec70c86e3e4 admin admin] The token cache is being invalidate because project 34e6a886fdef4899a6261200d9ecd640 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81577) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 30 02:02:29.882001 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-4485d043-5e4a-4d87-8514-4ec70c86e3e4 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:02:29.888297 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-4485d043-5e4a-4d87-8514-4ec70c86e3e4 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:02:29.888297 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 899/1784] 10.222.0.27 () {58 vars in 1233 bytes} [Tue Aug 30 02:02:29 2022] DELETE /identity/v3/projects/34e6a886fdef4899a6261200d9ecd640 => generated 0 bytes in 515 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 02:02:29.888750 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 886/1785] 10.222.0.27 () {60 vars in 1153 bytes} [Tue Aug 30 02:02:29 2022] POST /identity/v3/projects => generated 371 bytes in 439 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 02:02:29.891705 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc57198a-7011-458c-9c1b-4eeb27b59ce9 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:02:29.891967 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc57198a-7011-458c-9c1b-4eeb27b59ce9 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:02:29.892219 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3166d378-a7b8-4da9-bcdb-9b76389a4a51 None None] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:02:29.892363 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc57198a-7011-458c-9c1b-4eeb27b59ce9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:02:29.892787 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3166d378-a7b8-4da9-bcdb-9b76389a4a51 None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:02:29.897572 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3166d378-a7b8-4da9-bcdb-9b76389a4a51 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:02:29.924587 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.auth.core [None req-3166d378-a7b8-4da9-bcdb-9b76389a4a51 None None] MFA Rules not processed for user `d64f2e71361644589bec8a06e6eceed1`. Rule list: `[]` (Enabled: `True`). {{(pid=81578) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:02:29.926104 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-3166d378-a7b8-4da9-bcdb-9b76389a4a51 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:02:29.951295 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-cc57198a-7011-458c-9c1b-4eeb27b59ce9 None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:02:29.952559 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-cc57198a-7011-458c-9c1b-4eeb27b59ce9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:02:30.101965 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 887/1786] 10.222.0.27 () {58 vars in 1018 bytes} [Tue Aug 30 02:02:29 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 214 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 02:02:30.106022 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3166d378-a7b8-4da9-bcdb-9b76389a4a51 None None] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:02:30.111194 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-3166d378-a7b8-4da9-bcdb-9b76389a4a51 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:02:30.125148 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 900/1787] 10.222.0.27 () {58 vars in 958 bytes} [Tue Aug 30 02:02:29 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 238 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 02:02:30.127994 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc57198a-7011-458c-9c1b-4eeb27b59ce9 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:02:30.132966 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-cc57198a-7011-458c-9c1b-4eeb27b59ce9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:02:30.155928 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc57198a-7011-458c-9c1b-4eeb27b59ce9 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:02:30.164256 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3166d378-a7b8-4da9-bcdb-9b76389a4a51 None None] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:02:30.176222 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a04e1b36-9061-4ae4-91e2-e7ad08f55737 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:02:30.177281 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a04e1b36-9061-4ae4-91e2-e7ad08f55737 admin admin] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:02:30.177774 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a04e1b36-9061-4ae4-91e2-e7ad08f55737 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:02:30.178252 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a04e1b36-9061-4ae4-91e2-e7ad08f55737 admin admin] PATH_INFO: `/v3/users` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:02:30.179099 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a04e1b36-9061-4ae4-91e2-e7ad08f55737 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:02:30.181769 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2fa2f4f-ffd0-439b-bb64-0bf9425888fb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:02:30.182415 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2fa2f4f-ffd0-439b-bb64-0bf9425888fb admin admin] REQUEST_METHOD: `DELETE` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:02:30.182693 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2fa2f4f-ffd0-439b-bb64-0bf9425888fb admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:02:30.182958 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2fa2f4f-ffd0-439b-bb64-0bf9425888fb admin admin] PATH_INFO: `/v3/users/ca58bd7727d44d3594694c951ed88bcd` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:02:30.184426 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a04e1b36-9061-4ae4-91e2-e7ad08f55737 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:02:30.189710 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2fa2f4f-ffd0-439b-bb64-0bf9425888fb admin admin] RBAC: Authorizing `identity:delete_user(user_id=ca58bd7727d44d3594694c951ed88bcd)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:02:30.198784 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2fa2f4f-ffd0-439b-bb64-0bf9425888fb admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:02:30.312949 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 901/1788] 10.222.0.27 () {60 vars in 1144 bytes} [Tue Aug 30 02:02:30 2022] POST /identity/v3/users => generated 292 bytes in 185 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 02:02:30.314644 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a04e1b36-9061-4ae4-91e2-e7ad08f55737 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:02:30.363866 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a04e1b36-9061-4ae4-91e2-e7ad08f55737 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:02:30.388538 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d504464-934f-45a8-830e-2b499c590c20 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:02:30.389178 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d504464-934f-45a8-830e-2b499c590c20 admin admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:02:30.389482 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d504464-934f-45a8-830e-2b499c590c20 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:02:30.389756 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d504464-934f-45a8-830e-2b499c590c20 admin admin] PATH_INFO: `/v3/domains` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:02:30.397346 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d504464-934f-45a8-830e-2b499c590c20 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 02:02:30.397346 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d504464-934f-45a8-830e-2b499c590c20 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:02:30.397346 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d504464-934f-45a8-830e-2b499c590c20 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:02:30.423649 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 902/1789] 10.222.0.27 () {58 vars in 1153 bytes} [Tue Aug 30 02:02:30 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 109 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 02:02:30.424797 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d504464-934f-45a8-830e-2b499c590c20 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:02:30.452168 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d504464-934f-45a8-830e-2b499c590c20 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:02:30.463815 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00e75778-7855-43c4-a298-31610eed15c0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:02:30.464438 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00e75778-7855-43c4-a298-31610eed15c0 admin admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:02:30.464785 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00e75778-7855-43c4-a298-31610eed15c0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:02:30.465109 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00e75778-7855-43c4-a298-31610eed15c0 admin admin] PATH_INFO: `/v3/roles` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:02:30.465998 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00e75778-7855-43c4-a298-31610eed15c0 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:02:30.490424 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00e75778-7855-43c4-a298-31610eed15c0 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:02:30.516357 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 903/1790] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Aug 30 02:02:30 2022] GET /identity/v3/roles => generated 1420 bytes in 92 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 02:02:30.517448 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00e75778-7855-43c4-a298-31610eed15c0 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:02:30.550678 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00e75778-7855-43c4-a298-31610eed15c0 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:02:30.556032 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2e47a27-b6e0-4e38-bb24-859c9773f0d1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:02:30.568483 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2e47a27-b6e0-4e38-bb24-859c9773f0d1 admin admin] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:02:30.571382 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2e47a27-b6e0-4e38-bb24-859c9773f0d1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:02:30.571382 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2e47a27-b6e0-4e38-bb24-859c9773f0d1 admin admin] PATH_INFO: `/v3/projects` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:02:30.571382 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2e47a27-b6e0-4e38-bb24-859c9773f0d1 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:02:30.572472 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2e47a27-b6e0-4e38-bb24-859c9773f0d1 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:02:30.585112 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-d2fa2f4f-ffd0-439b-bb64-0bf9425888fb admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'ca58bd7727d44d3594694c951ed88bcd'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:02:30.587160 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-d2fa2f4f-ffd0-439b-bb64-0bf9425888fb admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'ca58bd7727d44d3594694c951ed88bcd'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:02:30.746013 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-d2fa2f4f-ffd0-439b-bb64-0bf9425888fb admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'ca58bd7727d44d3594694c951ed88bcd'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:02:30.751226 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-d2fa2f4f-ffd0-439b-bb64-0bf9425888fb admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'ca58bd7727d44d3594694c951ed88bcd'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:02:30.777706 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 904/1791] 10.222.0.27 () {60 vars in 1153 bytes} [Tue Aug 30 02:02:30 2022] POST /identity/v3/projects => generated 383 bytes in 261 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 02:02:30.778781 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2e47a27-b6e0-4e38-bb24-859c9773f0d1 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:02:30.786686 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-d2fa2f4f-ffd0-439b-bb64-0bf9425888fb admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'ca58bd7727d44d3594694c951ed88bcd'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:02:30.787122 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-e2e47a27-b6e0-4e38-bb24-859c9773f0d1 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:02:30.811910 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 888/1792] 10.222.0.27 () {58 vars in 1224 bytes} [Tue Aug 30 02:02:30 2022] DELETE /identity/v3/users/ca58bd7727d44d3594694c951ed88bcd => generated 0 bytes in 709 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 02:02:30.813304 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2fa2f4f-ffd0-439b-bb64-0bf9425888fb admin admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:02:30.821090 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-d2fa2f4f-ffd0-439b-bb64-0bf9425888fb admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:02:30.957675 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2e47a27-b6e0-4e38-bb24-859c9773f0d1 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:02:31.064090 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2fa2f4f-ffd0-439b-bb64-0bf9425888fb admin admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:02:31.096772 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11a4cd61-9898-43af-a945-8a497a7dc441 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:02:31.102331 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11a4cd61-9898-43af-a945-8a497a7dc441 admin admin] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:02:31.102631 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11a4cd61-9898-43af-a945-8a497a7dc441 admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:02:31.102911 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11a4cd61-9898-43af-a945-8a497a7dc441 admin admin] PATH_INFO: `/v3/users` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:02:31.104764 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11a4cd61-9898-43af-a945-8a497a7dc441 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:02:31.108873 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11a4cd61-9898-43af-a945-8a497a7dc441 admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:02:31.168356 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-731a649b-c8ed-4f34-bed4-0859681c4096 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:02:31.169919 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-731a649b-c8ed-4f34-bed4-0859681c4096 admin admin] REQUEST_METHOD: `PUT` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:02:31.170442 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-731a649b-c8ed-4f34-bed4-0859681c4096 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:02:31.170824 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-731a649b-c8ed-4f34-bed4-0859681c4096 admin admin] PATH_INFO: `/v3/projects/264f8a2690f64ce6abcf196780b73b9b/users/43136d6e83234361b054b7592f87aaa4/roles/ead5fac86a0747ea8823a82efcea8ed7` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:02:31.191610 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-731a649b-c8ed-4f34-bed4-0859681c4096 admin admin] RBAC: Authorizing `identity:create_grant(project_id=264f8a2690f64ce6abcf196780b73b9b, user_id=43136d6e83234361b054b7592f87aaa4, role_id=ead5fac86a0747ea8823a82efcea8ed7)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:02:31.195493 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-731a649b-c8ed-4f34-bed4-0859681c4096 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:02:31.256507 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 889/1793] 10.222.0.27 () {60 vars in 1144 bytes} [Tue Aug 30 02:02:30 2022] POST /identity/v3/users => generated 292 bytes in 444 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 02:02:31.258176 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11a4cd61-9898-43af-a945-8a497a7dc441 admin admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:02:31.260611 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-11a4cd61-9898-43af-a945-8a497a7dc441 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:02:31.329332 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 905/1794] 10.222.0.27 () {60 vars in 1483 bytes} [Tue Aug 30 02:02:30 2022] PUT /identity/v3/projects/264f8a2690f64ce6abcf196780b73b9b/users/43136d6e83234361b054b7592f87aaa4/roles/ead5fac86a0747ea8823a82efcea8ed7 => generated 0 bytes in 551 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 02:02:31.333993 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-731a649b-c8ed-4f34-bed4-0859681c4096 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:02:31.486836 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11a4cd61-9898-43af-a945-8a497a7dc441 admin admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:02:31.496920 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-731a649b-c8ed-4f34-bed4-0859681c4096 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:02:31.520576 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c3f44e2-9dac-4b66-9e48-11cf3e30613a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:02:31.520576 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c3f44e2-9dac-4b66-9e48-11cf3e30613a admin admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:02:31.520576 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c3f44e2-9dac-4b66-9e48-11cf3e30613a admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:02:31.520576 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c3f44e2-9dac-4b66-9e48-11cf3e30613a admin admin] PATH_INFO: `/v3/roles` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:02:31.522595 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c3f44e2-9dac-4b66-9e48-11cf3e30613a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:02:31.528086 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c3f44e2-9dac-4b66-9e48-11cf3e30613a admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:02:31.575700 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 906/1795] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Aug 30 02:02:31 2022] GET /identity/v3/roles => generated 1420 bytes in 246 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 02:02:31.577036 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c3f44e2-9dac-4b66-9e48-11cf3e30613a admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:02:31.601140 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c3f44e2-9dac-4b66-9e48-11cf3e30613a admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:02:31.648752 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ed496a9-3214-4a88-9960-3ec811811db5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:02:31.652178 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ed496a9-3214-4a88-9960-3ec811811db5 admin admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:02:31.658328 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ed496a9-3214-4a88-9960-3ec811811db5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:02:31.658681 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ed496a9-3214-4a88-9960-3ec811811db5 admin admin] PATH_INFO: `/v3/roles` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:02:31.659759 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ed496a9-3214-4a88-9960-3ec811811db5 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:02:31.672157 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ed496a9-3214-4a88-9960-3ec811811db5 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:02:31.711306 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 907/1796] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Aug 30 02:02:31 2022] GET /identity/v3/roles => generated 1420 bytes in 135 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 02:02:31.729854 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ed496a9-3214-4a88-9960-3ec811811db5 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:02:31.752841 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b464b623-cf7e-49f0-8347-82094b90d413 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd64f2e71361644589bec8a06e6eceed1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4562f3951fbe465280073e60a11e44e2', 'project_domain_id': 'default', 'roles': ['member', 'service', 'admin', 'ResellerAdmin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:02:31.754076 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b464b623-cf7e-49f0-8347-82094b90d413 service nova] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:02:31.756041 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b464b623-cf7e-49f0-8347-82094b90d413 service nova] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:02:31.756785 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b464b623-cf7e-49f0-8347-82094b90d413 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:02:31.759781 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-b464b623-cf7e-49f0-8347-82094b90d413 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:02:31.769924 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ed496a9-3214-4a88-9960-3ec811811db5 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:02:31.784841 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b464b623-cf7e-49f0-8347-82094b90d413 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:02:31.794473 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b464b623-cf7e-49f0-8347-82094b90d413 service nova] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:02:31.794859 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6da97e43-925a-48f7-9aaf-205ccd331599 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:02:31.794859 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6da97e43-925a-48f7-9aaf-205ccd331599 admin admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:02:31.794859 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6da97e43-925a-48f7-9aaf-205ccd331599 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:02:31.794859 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6da97e43-925a-48f7-9aaf-205ccd331599 admin admin] PATH_INFO: `/v3/roles` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:02:31.795293 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6da97e43-925a-48f7-9aaf-205ccd331599 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:02:31.795293 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6da97e43-925a-48f7-9aaf-205ccd331599 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:02:31.817608 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 908/1797] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Aug 30 02:02:31 2022] GET /identity/v3/roles => generated 1420 bytes in 97 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 02:02:31.817608 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6da97e43-925a-48f7-9aaf-205ccd331599 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:02:31.820934 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 890/1798] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Aug 30 02:02:31 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 564 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 02:02:31.822950 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b464b623-cf7e-49f0-8347-82094b90d413 service nova] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:02:31.841647 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6da97e43-925a-48f7-9aaf-205ccd331599 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:02:31.849435 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b464b623-cf7e-49f0-8347-82094b90d413 service nova] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:02:31.866821 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cacb6ab9-e547-40c7-9054-011323901f2e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:02:31.867952 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cacb6ab9-e547-40c7-9054-011323901f2e admin admin] REQUEST_METHOD: `PUT` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:02:31.868310 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cacb6ab9-e547-40c7-9054-011323901f2e admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:02:31.868715 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cacb6ab9-e547-40c7-9054-011323901f2e admin admin] PATH_INFO: `/v3/projects/264f8a2690f64ce6abcf196780b73b9b/users/43136d6e83234361b054b7592f87aaa4/roles/ead5fac86a0747ea8823a82efcea8ed7` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:02:31.872945 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5deee16-d8e3-4058-8d16-d1e1d2f5ff43 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:02:31.878253 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5deee16-d8e3-4058-8d16-d1e1d2f5ff43 admin admin] REQUEST_METHOD: `PUT` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:02:31.878378 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cacb6ab9-e547-40c7-9054-011323901f2e admin admin] RBAC: Authorizing `identity:create_grant(project_id=264f8a2690f64ce6abcf196780b73b9b, user_id=43136d6e83234361b054b7592f87aaa4, role_id=ead5fac86a0747ea8823a82efcea8ed7)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:02:31.878458 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5deee16-d8e3-4058-8d16-d1e1d2f5ff43 admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:02:31.878458 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5deee16-d8e3-4058-8d16-d1e1d2f5ff43 admin admin] PATH_INFO: `/v3/projects/c18e9b4b4e6542e1a7d3e08f6acea712/users/1a7ae71307f642aaa19715f54d72b2ff/roles/1fb81afcb9f245f3b989add94598a063` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:02:31.881444 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cacb6ab9-e547-40c7-9054-011323901f2e admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:02:31.929965 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 909/1799] 10.222.0.27 () {60 vars in 1483 bytes} [Tue Aug 30 02:02:31 2022] PUT /identity/v3/projects/264f8a2690f64ce6abcf196780b73b9b/users/43136d6e83234361b054b7592f87aaa4/roles/ead5fac86a0747ea8823a82efcea8ed7 => generated 0 bytes in 105 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 02:02:31.931864 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b240eab-bea0-45ed-9c04-03c1f3a27d2c None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:02:31.932160 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b240eab-bea0-45ed-9c04-03c1f3a27d2c None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:02:31.932422 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b240eab-bea0-45ed-9c04-03c1f3a27d2c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:02:31.950573 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5deee16-d8e3-4058-8d16-d1e1d2f5ff43 admin admin] RBAC: Authorizing `identity:create_grant(project_id=c18e9b4b4e6542e1a7d3e08f6acea712, user_id=1a7ae71307f642aaa19715f54d72b2ff, role_id=1fb81afcb9f245f3b989add94598a063)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:02:31.953437 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5deee16-d8e3-4058-8d16-d1e1d2f5ff43 admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:02:32.018823 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-7b240eab-bea0-45ed-9c04-03c1f3a27d2c None None] MFA Rules not processed for user `43136d6e83234361b054b7592f87aaa4`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:02:32.020125 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-7b240eab-bea0-45ed-9c04-03c1f3a27d2c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:02:32.058707 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 891/1800] 10.222.0.27 () {60 vars in 1483 bytes} [Tue Aug 30 02:02:31 2022] PUT /identity/v3/projects/c18e9b4b4e6542e1a7d3e08f6acea712/users/1a7ae71307f642aaa19715f54d72b2ff/roles/1fb81afcb9f245f3b989add94598a063 => generated 0 bytes in 237 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 02:02:32.059965 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5deee16-d8e3-4058-8d16-d1e1d2f5ff43 admin admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:02:32.075760 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-a5deee16-d8e3-4058-8d16-d1e1d2f5ff43 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:02:32.256348 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5deee16-d8e3-4058-8d16-d1e1d2f5ff43 admin admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:02:32.264270 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4eae73d6-35e4-456b-ac3d-bbc82b7ea52e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:02:32.264995 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4eae73d6-35e4-456b-ac3d-bbc82b7ea52e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:02:32.266298 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4eae73d6-35e4-456b-ac3d-bbc82b7ea52e admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:02:32.266835 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4eae73d6-35e4-456b-ac3d-bbc82b7ea52e admin admin] PATH_INFO: `/v3/projects/2d2e26f8cce64a02929ead6ed540d36d` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:02:32.269146 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4eae73d6-35e4-456b-ac3d-bbc82b7ea52e admin admin] RBAC: Authorizing `identity:delete_project(project_id=2d2e26f8cce64a02929ead6ed540d36d)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:02:32.273484 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4eae73d6-35e4-456b-ac3d-bbc82b7ea52e admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:02:32.322515 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 910/1801] 10.222.0.27 () {58 vars in 958 bytes} [Tue Aug 30 02:02:31 2022] POST /identity/v3/auth/tokens => generated 3315 bytes in 391 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 30 02:02:32.326008 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9bcb17d-7ee7-4e2e-81e0-0660cd4a5463 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:02:32.326464 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9bcb17d-7ee7-4e2e-81e0-0660cd4a5463 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:02:32.326777 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9bcb17d-7ee7-4e2e-81e0-0660cd4a5463 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:02:32.438186 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-b9bcb17d-7ee7-4e2e-81e0-0660cd4a5463 None None] MFA Rules not processed for user `1a7ae71307f642aaa19715f54d72b2ff`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:02:32.441214 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-b9bcb17d-7ee7-4e2e-81e0-0660cd4a5463 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:02:32.530021 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 911/1802] 10.222.0.27 () {58 vars in 958 bytes} [Tue Aug 30 02:02:32 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 207 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 02:02:32.612226 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9bcb17d-7ee7-4e2e-81e0-0660cd4a5463 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:02:32.612226 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-b9bcb17d-7ee7-4e2e-81e0-0660cd4a5463 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:02:32.612226 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9bcb17d-7ee7-4e2e-81e0-0660cd4a5463 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:02:32.612511 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9be9d8a-2f37-4910-96f1-f5ede4ae28ee service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ab936b3767a8417aa80c8c8f1b84b667', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4562f3951fbe465280073e60a11e44e2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:02:32.615443 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9be9d8a-2f37-4910-96f1-f5ede4ae28ee service cinder] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:02:32.615813 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9be9d8a-2f37-4910-96f1-f5ede4ae28ee service cinder] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:02:32.616164 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9be9d8a-2f37-4910-96f1-f5ede4ae28ee service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:02:32.750558 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9be9d8a-2f37-4910-96f1-f5ede4ae28ee service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:02:32.751999 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9be9d8a-2f37-4910-96f1-f5ede4ae28ee service cinder] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:02:32.779216 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 912/1803] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Aug 30 02:02:32 2022] GET /identity/v3/auth/tokens => generated 3315 bytes in 248 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 02:02:32.779940 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9be9d8a-2f37-4910-96f1-f5ede4ae28ee service cinder] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:02:32.805806 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9be9d8a-2f37-4910-96f1-f5ede4ae28ee service cinder] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:02:32.813138 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b2a2caa-feb8-45cc-98d7-832cd31c73b2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:02:32.813789 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b2a2caa-feb8-45cc-98d7-832cd31c73b2 admin admin] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:02:32.814116 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b2a2caa-feb8-45cc-98d7-832cd31c73b2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:02:32.814707 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b2a2caa-feb8-45cc-98d7-832cd31c73b2 admin admin] PATH_INFO: `/v3/projects` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:02:32.815414 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b2a2caa-feb8-45cc-98d7-832cd31c73b2 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:02:32.818933 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b2a2caa-feb8-45cc-98d7-832cd31c73b2 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:02:32.901277 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-4eae73d6-35e4-456b-ac3d-bbc82b7ea52e admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '2d2e26f8cce64a02929ead6ed540d36d', 'request_id': 'req-4eae73d6-35e4-456b-ac3d-bbc82b7ea52e'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:02:32.901277 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-4eae73d6-35e4-456b-ac3d-bbc82b7ea52e admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '2d2e26f8cce64a02929ead6ed540d36d', 'request_id': 'req-4eae73d6-35e4-456b-ac3d-bbc82b7ea52e'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:02:32.910599 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-4eae73d6-35e4-456b-ac3d-bbc82b7ea52e admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '2d2e26f8cce64a02929ead6ed540d36d', 'request_id': 'req-4eae73d6-35e4-456b-ac3d-bbc82b7ea52e'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:02:33.452978 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 913/1804] 10.222.0.27 () {60 vars in 1153 bytes} [Tue Aug 30 02:02:32 2022] POST /identity/v3/projects => generated 385 bytes in 674 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 02:02:33.461937 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b2a2caa-feb8-45cc-98d7-832cd31c73b2 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:02:33.500209 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-4eae73d6-35e4-456b-ac3d-bbc82b7ea52e admin admin] The token cache is being invalidate because project 2d2e26f8cce64a02929ead6ed540d36d was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81578) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 30 02:02:33.500821 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-4eae73d6-35e4-456b-ac3d-bbc82b7ea52e admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:02:33.502478 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-4eae73d6-35e4-456b-ac3d-bbc82b7ea52e admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:02:33.504548 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 892/1805] 10.222.0.27 () {58 vars in 1233 bytes} [Tue Aug 30 02:02:32 2022] DELETE /identity/v3/projects/2d2e26f8cce64a02929ead6ed540d36d => generated 0 bytes in 1445 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 02:02:33.509523 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bace4310-7493-47c8-afab-e199c4d57b6d None None] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:02:33.510095 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bace4310-7493-47c8-afab-e199c4d57b6d None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:02:33.510585 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bace4310-7493-47c8-afab-e199c4d57b6d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:02:33.546723 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.auth.core [None req-bace4310-7493-47c8-afab-e199c4d57b6d None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81578) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:02:33.548630 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-bace4310-7493-47c8-afab-e199c4d57b6d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:02:33.634697 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b2a2caa-feb8-45cc-98d7-832cd31c73b2 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:02:33.643787 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46d0c0a8-0033-475f-b7d2-3d199543b227 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:02:33.643787 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46d0c0a8-0033-475f-b7d2-3d199543b227 admin admin] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:02:33.655381 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46d0c0a8-0033-475f-b7d2-3d199543b227 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:02:33.655731 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46d0c0a8-0033-475f-b7d2-3d199543b227 admin admin] PATH_INFO: `/v3/users` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:02:33.657592 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46d0c0a8-0033-475f-b7d2-3d199543b227 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:02:33.662086 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46d0c0a8-0033-475f-b7d2-3d199543b227 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:02:33.737913 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 893/1806] 10.222.0.27 () {58 vars in 958 bytes} [Tue Aug 30 02:02:33 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 224 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 02:02:33.741175 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e40e652-371f-46e4-a4e7-e9b313fbe7f5 None None] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:02:33.741175 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e40e652-371f-46e4-a4e7-e9b313fbe7f5 None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:02:33.741175 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e40e652-371f-46e4-a4e7-e9b313fbe7f5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:02:33.787353 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.auth.core [None req-5e40e652-371f-46e4-a4e7-e9b313fbe7f5 None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81578) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:02:33.790023 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-5e40e652-371f-46e4-a4e7-e9b313fbe7f5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:02:33.806451 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 894/1807] 10.222.0.27 () {58 vars in 958 bytes} [Tue Aug 30 02:02:33 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 68 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 30 02:02:33.808114 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e40e652-371f-46e4-a4e7-e9b313fbe7f5 None None] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:02:33.825998 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e40e652-371f-46e4-a4e7-e9b313fbe7f5 None None] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:02:33.831523 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b580103b-e604-4f3e-bb69-9f185065a31f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:02:33.833610 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b580103b-e604-4f3e-bb69-9f185065a31f admin admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:02:33.833610 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b580103b-e604-4f3e-bb69-9f185065a31f admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:02:33.833610 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b580103b-e604-4f3e-bb69-9f185065a31f admin admin] PATH_INFO: `/v3/domains` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:02:33.833610 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b580103b-e604-4f3e-bb69-9f185065a31f admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81578) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 02:02:33.834054 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b580103b-e604-4f3e-bb69-9f185065a31f admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:02:33.835410 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 914/1808] 10.222.0.27 () {60 vars in 1144 bytes} [Tue Aug 30 02:02:33 2022] POST /identity/v3/users => generated 293 bytes in 374 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 02:02:33.835715 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b580103b-e604-4f3e-bb69-9f185065a31f admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:02:33.839632 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46d0c0a8-0033-475f-b7d2-3d199543b227 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:02:33.843322 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-46d0c0a8-0033-475f-b7d2-3d199543b227 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:02:33.857734 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 895/1809] 10.222.0.27 () {58 vars in 1153 bytes} [Tue Aug 30 02:02:33 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 50 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 02:02:33.861726 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b580103b-e604-4f3e-bb69-9f185065a31f admin admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:02:33.875930 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b580103b-e604-4f3e-bb69-9f185065a31f admin admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:02:33.877740 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46d0c0a8-0033-475f-b7d2-3d199543b227 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:02:33.882340 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-288198b1-fc34-4249-bd4c-1dbaf16e5516 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:02:33.883444 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-288198b1-fc34-4249-bd4c-1dbaf16e5516 admin admin] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:02:33.883901 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-288198b1-fc34-4249-bd4c-1dbaf16e5516 admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:02:33.884362 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-288198b1-fc34-4249-bd4c-1dbaf16e5516 admin admin] PATH_INFO: `/v3/projects` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:02:33.885587 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-288198b1-fc34-4249-bd4c-1dbaf16e5516 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:02:33.890427 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-288198b1-fc34-4249-bd4c-1dbaf16e5516 admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:02:33.890565 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-466e8aca-7d3a-44cf-bca8-1fca998d7af1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:02:33.900735 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-466e8aca-7d3a-44cf-bca8-1fca998d7af1 admin admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:02:33.902305 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-466e8aca-7d3a-44cf-bca8-1fca998d7af1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:02:33.904306 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-466e8aca-7d3a-44cf-bca8-1fca998d7af1 admin admin] PATH_INFO: `/v3/roles` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:02:33.905241 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-466e8aca-7d3a-44cf-bca8-1fca998d7af1 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:02:33.910931 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-466e8aca-7d3a-44cf-bca8-1fca998d7af1 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:02:33.938353 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 915/1810] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Aug 30 02:02:33 2022] GET /identity/v3/roles => generated 1420 bytes in 99 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 02:02:33.942689 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-466e8aca-7d3a-44cf-bca8-1fca998d7af1 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:02:33.961728 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-466e8aca-7d3a-44cf-bca8-1fca998d7af1 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:02:33.978232 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-073691a7-eef9-48ed-bbf1-db8fbb96e9bf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:02:33.978906 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-073691a7-eef9-48ed-bbf1-db8fbb96e9bf admin admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:02:33.979119 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-073691a7-eef9-48ed-bbf1-db8fbb96e9bf admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:02:33.979394 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-073691a7-eef9-48ed-bbf1-db8fbb96e9bf admin admin] PATH_INFO: `/v3/roles` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:02:33.980102 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-073691a7-eef9-48ed-bbf1-db8fbb96e9bf admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:02:33.981826 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-073691a7-eef9-48ed-bbf1-db8fbb96e9bf admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:02:34.019310 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:02:34.026629 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 916/1811] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Aug 30 02:02:33 2022] GET /identity/v3/roles => generated 1420 bytes in 85 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 02:02:34.031445 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 896/1812] 10.222.0.27 () {60 vars in 1153 bytes} [Tue Aug 30 02:02:33 2022] POST /identity/v3/projects => generated 387 bytes in 173 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 02:02:34.035043 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-073691a7-eef9-48ed-bbf1-db8fbb96e9bf admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:02:34.036364 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-288198b1-fc34-4249-bd4c-1dbaf16e5516 admin admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:02:34.087063 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-073691a7-eef9-48ed-bbf1-db8fbb96e9bf admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:02:34.218646 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-288198b1-fc34-4249-bd4c-1dbaf16e5516 admin admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:02:34.236473 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e0c4fe1-61ad-4280-afa7-e55a7087f85a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:02:34.237992 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e0c4fe1-61ad-4280-afa7-e55a7087f85a admin admin] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:02:34.239298 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e0c4fe1-61ad-4280-afa7-e55a7087f85a admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:02:34.240418 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e0c4fe1-61ad-4280-afa7-e55a7087f85a admin admin] PATH_INFO: `/v3/users` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:02:34.241134 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e0c4fe1-61ad-4280-afa7-e55a7087f85a admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:02:34.248558 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e0c4fe1-61ad-4280-afa7-e55a7087f85a admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:02:34.333499 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff5f8213-62c8-401b-909e-874cd8568fb3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:02:34.333499 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff5f8213-62c8-401b-909e-874cd8568fb3 admin admin] REQUEST_METHOD: `PUT` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:02:34.333499 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff5f8213-62c8-401b-909e-874cd8568fb3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:02:34.333499 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff5f8213-62c8-401b-909e-874cd8568fb3 admin admin] PATH_INFO: `/v3/projects/74b66f9413bc48939877bad0a15cd509/users/c44667b3e57b4703943113b10b548d73/roles/1fb81afcb9f245f3b989add94598a063` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:02:34.362548 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 897/1813] 10.222.0.27 () {60 vars in 1144 bytes} [Tue Aug 30 02:02:34 2022] POST /identity/v3/users => generated 294 bytes in 326 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 02:02:34.368125 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff5f8213-62c8-401b-909e-874cd8568fb3 admin admin] RBAC: Authorizing `identity:create_grant(project_id=74b66f9413bc48939877bad0a15cd509, user_id=c44667b3e57b4703943113b10b548d73, role_id=1fb81afcb9f245f3b989add94598a063)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:02:34.368125 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff5f8213-62c8-401b-909e-874cd8568fb3 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:02:34.369215 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e0c4fe1-61ad-4280-afa7-e55a7087f85a admin admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:02:34.389114 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e0c4fe1-61ad-4280-afa7-e55a7087f85a admin admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:02:34.411901 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45a66ee6-6594-405f-994d-129c865b786e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:02:34.411901 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45a66ee6-6594-405f-994d-129c865b786e admin admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:02:34.412244 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45a66ee6-6594-405f-994d-129c865b786e admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:02:34.412244 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45a66ee6-6594-405f-994d-129c865b786e admin admin] PATH_INFO: `/v3/roles` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:02:34.413009 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45a66ee6-6594-405f-994d-129c865b786e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:02:34.416330 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45a66ee6-6594-405f-994d-129c865b786e admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:02:34.447398 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 898/1814] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Aug 30 02:02:34 2022] GET /identity/v3/roles => generated 1420 bytes in 79 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 30 02:02:34.451261 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45a66ee6-6594-405f-994d-129c865b786e admin admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:02:34.487973 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 917/1815] 10.222.0.27 () {60 vars in 1483 bytes} [Tue Aug 30 02:02:34 2022] PUT /identity/v3/projects/74b66f9413bc48939877bad0a15cd509/users/c44667b3e57b4703943113b10b548d73/roles/1fb81afcb9f245f3b989add94598a063 => generated 0 bytes in 453 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 02:02:34.497954 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45a66ee6-6594-405f-994d-129c865b786e admin admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:02:34.503981 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94e69a99-8fff-4f8f-8afc-6c3dda56a33a None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:02:34.504277 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94e69a99-8fff-4f8f-8afc-6c3dda56a33a None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:02:34.504531 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94e69a99-8fff-4f8f-8afc-6c3dda56a33a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:02:34.569041 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-94e69a99-8fff-4f8f-8afc-6c3dda56a33a None None] MFA Rules not processed for user `c44667b3e57b4703943113b10b548d73`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:02:34.572852 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-94e69a99-8fff-4f8f-8afc-6c3dda56a33a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:02:34.580981 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:02:34.633846 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-137fa068-f9ec-4578-9253-6f24c58b1b95 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:02:34.639673 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-137fa068-f9ec-4578-9253-6f24c58b1b95 admin admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:02:34.639673 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-137fa068-f9ec-4578-9253-6f24c58b1b95 admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:02:34.639973 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-137fa068-f9ec-4578-9253-6f24c58b1b95 admin admin] PATH_INFO: `/v3/roles` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:02:34.640704 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-137fa068-f9ec-4578-9253-6f24c58b1b95 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:02:34.642811 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-137fa068-f9ec-4578-9253-6f24c58b1b95 admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:02:34.687506 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 899/1816] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Aug 30 02:02:34 2022] GET /identity/v3/roles => generated 1420 bytes in 237 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 30 02:02:34.688710 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-137fa068-f9ec-4578-9253-6f24c58b1b95 admin admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:02:34.743743 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-137fa068-f9ec-4578-9253-6f24c58b1b95 admin admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:02:34.762752 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b262d555-fffd-4753-8c97-07bf4464b073 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:02:34.763527 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b262d555-fffd-4753-8c97-07bf4464b073 admin admin] REQUEST_METHOD: `PUT` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:02:34.763906 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b262d555-fffd-4753-8c97-07bf4464b073 admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:02:34.764308 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b262d555-fffd-4753-8c97-07bf4464b073 admin admin] PATH_INFO: `/v3/projects/bc6021ca1ab14ac684ad7b6d8fab1efa/users/111b7de24c9a4a079b79b9da5c66bb64/roles/1fb81afcb9f245f3b989add94598a063` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:02:34.790183 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b262d555-fffd-4753-8c97-07bf4464b073 admin admin] RBAC: Authorizing `identity:create_grant(project_id=bc6021ca1ab14ac684ad7b6d8fab1efa, user_id=111b7de24c9a4a079b79b9da5c66bb64, role_id=1fb81afcb9f245f3b989add94598a063)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:02:34.793131 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b262d555-fffd-4753-8c97-07bf4464b073 admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:02:34.815287 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 918/1817] 10.222.0.27 () {58 vars in 958 bytes} [Tue Aug 30 02:02:34 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 312 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 30 02:02:34.822133 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94e69a99-8fff-4f8f-8afc-6c3dda56a33a None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:02:34.835751 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94e69a99-8fff-4f8f-8afc-6c3dda56a33a None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:02:34.842767 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdf4c6e7-d9c2-4542-8715-97315b52976e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:02:34.843544 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdf4c6e7-d9c2-4542-8715-97315b52976e admin admin] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:02:34.843967 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdf4c6e7-d9c2-4542-8715-97315b52976e admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:02:34.844340 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdf4c6e7-d9c2-4542-8715-97315b52976e admin admin] PATH_INFO: `/v3/projects` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:02:34.846150 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fdf4c6e7-d9c2-4542-8715-97315b52976e admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:02:34.848388 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fdf4c6e7-d9c2-4542-8715-97315b52976e admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:02:34.884222 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 900/1818] 10.222.0.27 () {60 vars in 1483 bytes} [Tue Aug 30 02:02:34 2022] PUT /identity/v3/projects/bc6021ca1ab14ac684ad7b6d8fab1efa/users/111b7de24c9a4a079b79b9da5c66bb64/roles/1fb81afcb9f245f3b989add94598a063 => generated 0 bytes in 196 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 02:02:34.897223 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48a33799-d5d3-4cf9-8516-f223ce959368 None None] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:02:34.897627 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48a33799-d5d3-4cf9-8516-f223ce959368 None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:02:34.897975 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48a33799-d5d3-4cf9-8516-f223ce959368 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:02:34.963499 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.auth.core [None req-48a33799-d5d3-4cf9-8516-f223ce959368 None None] MFA Rules not processed for user `111b7de24c9a4a079b79b9da5c66bb64`. Rule list: `[]` (Enabled: `True`). {{(pid=81578) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:02:34.964697 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-48a33799-d5d3-4cf9-8516-f223ce959368 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:02:34.983308 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 919/1819] 10.222.0.27 () {60 vars in 1153 bytes} [Tue Aug 30 02:02:34 2022] POST /identity/v3/projects => generated 385 bytes in 163 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 02:02:34.984174 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdf4c6e7-d9c2-4542-8715-97315b52976e admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:02:34.988870 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-fdf4c6e7-d9c2-4542-8715-97315b52976e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:02:35.174712 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdf4c6e7-d9c2-4542-8715-97315b52976e admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:02:35.185996 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 901/1820] 10.222.0.27 () {58 vars in 958 bytes} [Tue Aug 30 02:02:34 2022] POST /identity/v3/auth/tokens => generated 3264 bytes in 290 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 02:02:35.187635 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48a33799-d5d3-4cf9-8516-f223ce959368 None None] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:02:35.194751 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d2c1bab-99fa-4998-bf61-1c8f9cb8ab33 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:02:35.195696 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d2c1bab-99fa-4998-bf61-1c8f9cb8ab33 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:02:35.197199 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d2c1bab-99fa-4998-bf61-1c8f9cb8ab33 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:02:35.197716 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d2c1bab-99fa-4998-bf61-1c8f9cb8ab33 admin admin] PATH_INFO: `/v3/users/4194355610df4d95bbcbd46b25426c63` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:02:35.200282 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d2c1bab-99fa-4998-bf61-1c8f9cb8ab33 admin admin] RBAC: Authorizing `identity:delete_user(user_id=4194355610df4d95bbcbd46b25426c63)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:02:35.207572 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d2c1bab-99fa-4998-bf61-1c8f9cb8ab33 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:02:35.358439 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48a33799-d5d3-4cf9-8516-f223ce959368 None None] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:02:35.376497 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-881947f5-0f97-43f2-ad81-69622fe8654f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:02:35.376497 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-881947f5-0f97-43f2-ad81-69622fe8654f admin admin] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:02:35.376497 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-881947f5-0f97-43f2-ad81-69622fe8654f admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:02:35.376497 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-881947f5-0f97-43f2-ad81-69622fe8654f admin admin] PATH_INFO: `/v3/users` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:02:35.377072 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-881947f5-0f97-43f2-ad81-69622fe8654f admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:02:35.386140 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-881947f5-0f97-43f2-ad81-69622fe8654f admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:02:35.485493 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-1d2c1bab-99fa-4998-bf61-1c8f9cb8ab33 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '4194355610df4d95bbcbd46b25426c63'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:02:35.486534 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-1d2c1bab-99fa-4998-bf61-1c8f9cb8ab33 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '4194355610df4d95bbcbd46b25426c63'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:02:35.487361 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-1d2c1bab-99fa-4998-bf61-1c8f9cb8ab33 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '4194355610df4d95bbcbd46b25426c63'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:02:35.487769 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 902/1821] 10.222.0.27 () {60 vars in 1144 bytes} [Tue Aug 30 02:02:35 2022] POST /identity/v3/users => generated 299 bytes in 301 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 02:02:35.488688 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-881947f5-0f97-43f2-ad81-69622fe8654f admin admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:02:35.495200 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-881947f5-0f97-43f2-ad81-69622fe8654f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:02:35.512716 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-1d2c1bab-99fa-4998-bf61-1c8f9cb8ab33 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '4194355610df4d95bbcbd46b25426c63'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:02:35.517791 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-881947f5-0f97-43f2-ad81-69622fe8654f admin admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:02:35.521955 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-1d2c1bab-99fa-4998-bf61-1c8f9cb8ab33 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '4194355610df4d95bbcbd46b25426c63'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:02:35.524434 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-666adf15-1240-49dd-8790-39e727682687 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:02:35.525234 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-666adf15-1240-49dd-8790-39e727682687 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:02:35.525698 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-666adf15-1240-49dd-8790-39e727682687 admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:02:35.526106 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-666adf15-1240-49dd-8790-39e727682687 admin admin] PATH_INFO: `/v3/users/a1614048b7d0464a9cdf7dec2c6be828` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:02:35.528529 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-666adf15-1240-49dd-8790-39e727682687 admin admin] RBAC: Authorizing `identity:delete_user(user_id=a1614048b7d0464a9cdf7dec2c6be828)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:02:35.530624 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-666adf15-1240-49dd-8790-39e727682687 admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:02:35.592656 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 920/1822] 10.222.0.27 () {58 vars in 1224 bytes} [Tue Aug 30 02:02:34 2022] DELETE /identity/v3/users/4194355610df4d95bbcbd46b25426c63 => generated 0 bytes in 605 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 02:02:35.592656 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d2c1bab-99fa-4998-bf61-1c8f9cb8ab33 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:02:35.594996 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-1d2c1bab-99fa-4998-bf61-1c8f9cb8ab33 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:02:35.800833 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d2c1bab-99fa-4998-bf61-1c8f9cb8ab33 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:02:35.800833 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-754fb155-6050-411a-9421-e139400fef81 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:02:35.800833 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-754fb155-6050-411a-9421-e139400fef81 admin admin] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:02:35.800833 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-754fb155-6050-411a-9421-e139400fef81 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:02:35.801195 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-754fb155-6050-411a-9421-e139400fef81 admin admin] PATH_INFO: `/v3/projects` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:02:35.801195 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-754fb155-6050-411a-9421-e139400fef81 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:02:35.801195 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-754fb155-6050-411a-9421-e139400fef81 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:02:35.856461 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-666adf15-1240-49dd-8790-39e727682687 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'a1614048b7d0464a9cdf7dec2c6be828'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:02:35.857173 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-666adf15-1240-49dd-8790-39e727682687 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'a1614048b7d0464a9cdf7dec2c6be828'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:02:35.863442 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 921/1823] 10.222.0.27 () {60 vars in 1153 bytes} [Tue Aug 30 02:02:35 2022] POST /identity/v3/projects => generated 385 bytes in 274 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 02:02:35.864355 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-754fb155-6050-411a-9421-e139400fef81 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:02:35.866073 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-754fb155-6050-411a-9421-e139400fef81 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:02:35.927429 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-666adf15-1240-49dd-8790-39e727682687 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'a1614048b7d0464a9cdf7dec2c6be828'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:02:35.928524 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-666adf15-1240-49dd-8790-39e727682687 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'a1614048b7d0464a9cdf7dec2c6be828'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:02:35.932731 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-754fb155-6050-411a-9421-e139400fef81 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:02:35.939199 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0bb396b-d76a-42d8-b4a1-b45d957cac7a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:02:35.939358 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=98,keystone:INSERT=1,keystone:DELETE=1 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:02:35.940190 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0bb396b-d76a-42d8-b4a1-b45d957cac7a admin admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:02:35.940474 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0bb396b-d76a-42d8-b4a1-b45d957cac7a admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:02:35.941141 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-666adf15-1240-49dd-8790-39e727682687 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'a1614048b7d0464a9cdf7dec2c6be828'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:02:35.941268 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0bb396b-d76a-42d8-b4a1-b45d957cac7a admin admin] PATH_INFO: `/v3/roles` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:02:35.946323 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0bb396b-d76a-42d8-b4a1-b45d957cac7a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:02:35.946323 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0bb396b-d76a-42d8-b4a1-b45d957cac7a admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:02:35.957093 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 922/1824] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Aug 30 02:02:35 2022] GET /identity/v3/roles => generated 1420 bytes in 93 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 02:02:35.958598 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d3a717c-1168-4a68-92d4-8dacf8f5d607 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:02:35.958961 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d3a717c-1168-4a68-92d4-8dacf8f5d607 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:02:35.959308 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d3a717c-1168-4a68-92d4-8dacf8f5d607 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:02:35.962218 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 903/1825] 10.222.0.27 () {58 vars in 1224 bytes} [Tue Aug 30 02:02:35 2022] DELETE /identity/v3/users/a1614048b7d0464a9cdf7dec2c6be828 => generated 0 bytes in 471 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 02:02:35.962218 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-666adf15-1240-49dd-8790-39e727682687 admin admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:02:35.962436 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-666adf15-1240-49dd-8790-39e727682687 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:02:35.984341 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-6d3a717c-1168-4a68-92d4-8dacf8f5d607 None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:02:35.985205 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-6d3a717c-1168-4a68-92d4-8dacf8f5d607 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:02:36.034333 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-666adf15-1240-49dd-8790-39e727682687 admin admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:02:36.039126 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-caf58e8b-9dd7-4ccc-a518-3ef1c461265c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:02:36.039728 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-caf58e8b-9dd7-4ccc-a518-3ef1c461265c admin admin] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:02:36.040044 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-caf58e8b-9dd7-4ccc-a518-3ef1c461265c admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:02:36.040336 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-caf58e8b-9dd7-4ccc-a518-3ef1c461265c admin admin] PATH_INFO: `/v3/users` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:02:36.040959 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-caf58e8b-9dd7-4ccc-a518-3ef1c461265c admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:02:36.043403 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-caf58e8b-9dd7-4ccc-a518-3ef1c461265c admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:02:36.048577 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 923/1826] 10.222.0.27 () {58 vars in 958 bytes} [Tue Aug 30 02:02:35 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 02:02:36.049396 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d3a717c-1168-4a68-92d4-8dacf8f5d607 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:02:36.051035 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-6d3a717c-1168-4a68-92d4-8dacf8f5d607 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:02:36.064488 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d3a717c-1168-4a68-92d4-8dacf8f5d607 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:02:36.069429 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ea60420-a831-43a0-a256-da595dfa4b16 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:02:36.070127 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ea60420-a831-43a0-a256-da595dfa4b16 admin admin] REQUEST_METHOD: `PUT` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:02:36.070412 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ea60420-a831-43a0-a256-da595dfa4b16 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:02:36.070689 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ea60420-a831-43a0-a256-da595dfa4b16 admin admin] PATH_INFO: `/v3/projects/dfd11c6fe40545d3b647cafeedf448e8/users/d91a28c01d4f4ce1bc8b011e0d142d34/roles/ead5fac86a0747ea8823a82efcea8ed7` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:02:36.087683 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ea60420-a831-43a0-a256-da595dfa4b16 admin admin] RBAC: Authorizing `identity:create_grant(project_id=dfd11c6fe40545d3b647cafeedf448e8, user_id=d91a28c01d4f4ce1bc8b011e0d142d34, role_id=ead5fac86a0747ea8823a82efcea8ed7)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:02:36.090818 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ea60420-a831-43a0-a256-da595dfa4b16 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:02:36.143883 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 904/1827] 10.222.0.27 () {60 vars in 1144 bytes} [Tue Aug 30 02:02:35 2022] POST /identity/v3/users => generated 299 bytes in 184 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 02:02:36.145188 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f20ffa67-fa82-4837-891a-d436e592e0e0 None None] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:02:36.145492 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f20ffa67-fa82-4837-891a-d436e592e0e0 None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:02:36.145913 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f20ffa67-fa82-4837-891a-d436e592e0e0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:02:36.165894 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.auth.core [None req-f20ffa67-fa82-4837-891a-d436e592e0e0 None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81578) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:02:36.168686 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-f20ffa67-fa82-4837-891a-d436e592e0e0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:02:36.172873 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 924/1828] 10.222.0.27 () {60 vars in 1483 bytes} [Tue Aug 30 02:02:36 2022] PUT /identity/v3/projects/dfd11c6fe40545d3b647cafeedf448e8/users/d91a28c01d4f4ce1bc8b011e0d142d34/roles/ead5fac86a0747ea8823a82efcea8ed7 => generated 0 bytes in 123 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 02:02:36.173702 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ea60420-a831-43a0-a256-da595dfa4b16 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:02:36.176873 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-6ea60420-a831-43a0-a256-da595dfa4b16 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:02:36.240482 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 905/1829] 10.222.0.27 () {58 vars in 958 bytes} [Tue Aug 30 02:02:36 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 02:02:36.241247 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f20ffa67-fa82-4837-891a-d436e592e0e0 None None] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:02:36.250194 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ea60420-a831-43a0-a256-da595dfa4b16 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:02:36.251294 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f20ffa67-fa82-4837-891a-d436e592e0e0 None None] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:02:36.253833 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4d71ad8-7ab3-4b67-87e3-fa77f8a0f961 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f3ea1580c56f4814a92bbb2acf542905', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4562f3951fbe465280073e60a11e44e2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:02:36.254468 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4d71ad8-7ab3-4b67-87e3-fa77f8a0f961 service neutron] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:02:36.254744 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4d71ad8-7ab3-4b67-87e3-fa77f8a0f961 service neutron] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:02:36.255012 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4d71ad8-7ab3-4b67-87e3-fa77f8a0f961 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:02:36.256205 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d7c5301-4a03-4aa3-8d0c-a06440244484 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:02:36.256857 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d7c5301-4a03-4aa3-8d0c-a06440244484 admin admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:02:36.257150 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d7c5301-4a03-4aa3-8d0c-a06440244484 admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:02:36.257406 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d7c5301-4a03-4aa3-8d0c-a06440244484 admin admin] PATH_INFO: `/v3/roles` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:02:36.258075 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d7c5301-4a03-4aa3-8d0c-a06440244484 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:02:36.259681 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d7c5301-4a03-4aa3-8d0c-a06440244484 admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:02:36.264671 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4d71ad8-7ab3-4b67-87e3-fa77f8a0f961 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:02:36.266320 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4d71ad8-7ab3-4b67-87e3-fa77f8a0f961 service neutron] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:02:36.269968 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 906/1830] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Aug 30 02:02:36 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 02:02:36.270783 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d7c5301-4a03-4aa3-8d0c-a06440244484 admin admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:02:36.276650 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 925/1831] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Aug 30 02:02:36 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 103 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 02:02:36.278263 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4d71ad8-7ab3-4b67-87e3-fa77f8a0f961 service neutron] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:02:36.281399 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d7c5301-4a03-4aa3-8d0c-a06440244484 admin admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:02:36.287375 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4d71ad8-7ab3-4b67-87e3-fa77f8a0f961 service neutron] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:02:36.292563 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abaadec4-2c8d-42a2-afdb-e49da813fdfd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:02:36.292563 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abaadec4-2c8d-42a2-afdb-e49da813fdfd admin admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:02:36.292563 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abaadec4-2c8d-42a2-afdb-e49da813fdfd admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:02:36.292563 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abaadec4-2c8d-42a2-afdb-e49da813fdfd admin admin] PATH_INFO: `/v3/roles` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:02:36.293192 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abaadec4-2c8d-42a2-afdb-e49da813fdfd admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:02:36.293192 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abaadec4-2c8d-42a2-afdb-e49da813fdfd admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:02:36.293334 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1138fcac-4f2b-4ce1-aefe-edc575ac940d service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f3ea1580c56f4814a92bbb2acf542905', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4562f3951fbe465280073e60a11e44e2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:02:36.293334 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1138fcac-4f2b-4ce1-aefe-edc575ac940d service neutron] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:02:36.293807 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1138fcac-4f2b-4ce1-aefe-edc575ac940d service neutron] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:02:36.294161 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1138fcac-4f2b-4ce1-aefe-edc575ac940d service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:02:36.304006 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1138fcac-4f2b-4ce1-aefe-edc575ac940d service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:02:36.305670 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1138fcac-4f2b-4ce1-aefe-edc575ac940d service neutron] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:02:36.309664 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 907/1832] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Aug 30 02:02:36 2022] GET /identity/v3/roles => generated 1420 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 02:02:36.311585 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abaadec4-2c8d-42a2-afdb-e49da813fdfd admin admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:02:36.317130 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 926/1833] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Aug 30 02:02:36 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 40 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 02:02:36.318007 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1138fcac-4f2b-4ce1-aefe-edc575ac940d service neutron] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:02:36.325853 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abaadec4-2c8d-42a2-afdb-e49da813fdfd admin admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:02:36.329751 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1138fcac-4f2b-4ce1-aefe-edc575ac940d service neutron] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:02:36.333171 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27a66706-7015-4595-8c4a-f9a0505e8816 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:02:36.333171 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27a66706-7015-4595-8c4a-f9a0505e8816 admin admin] REQUEST_METHOD: `PUT` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:02:36.333171 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27a66706-7015-4595-8c4a-f9a0505e8816 admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:02:36.333171 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27a66706-7015-4595-8c4a-f9a0505e8816 admin admin] PATH_INFO: `/v3/projects/cfde2d9f016e46839128fe2158629c25/users/70f80cd5ef854ad6b7e8f13c18f68864/roles/ead5fac86a0747ea8823a82efcea8ed7` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:02:36.336726 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5187d6ef-8c3f-4069-8399-4799e9e6443b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:02:36.337738 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5187d6ef-8c3f-4069-8399-4799e9e6443b admin admin] REQUEST_METHOD: `PUT` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:02:36.338021 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5187d6ef-8c3f-4069-8399-4799e9e6443b admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:02:36.338299 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5187d6ef-8c3f-4069-8399-4799e9e6443b admin admin] PATH_INFO: `/v3/projects/dfd11c6fe40545d3b647cafeedf448e8/users/d91a28c01d4f4ce1bc8b011e0d142d34/roles/ead5fac86a0747ea8823a82efcea8ed7` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:02:36.340832 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5187d6ef-8c3f-4069-8399-4799e9e6443b admin admin] RBAC: Authorizing `identity:create_grant(project_id=dfd11c6fe40545d3b647cafeedf448e8, user_id=d91a28c01d4f4ce1bc8b011e0d142d34, role_id=ead5fac86a0747ea8823a82efcea8ed7)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:02:36.343900 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5187d6ef-8c3f-4069-8399-4799e9e6443b admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:02:36.352038 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27a66706-7015-4595-8c4a-f9a0505e8816 admin admin] RBAC: Authorizing `identity:create_grant(project_id=cfde2d9f016e46839128fe2158629c25, user_id=70f80cd5ef854ad6b7e8f13c18f68864, role_id=ead5fac86a0747ea8823a82efcea8ed7)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:02:36.355525 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 927/1834] 10.222.0.27 () {60 vars in 1483 bytes} [Tue Aug 30 02:02:36 2022] PUT /identity/v3/projects/dfd11c6fe40545d3b647cafeedf448e8/users/d91a28c01d4f4ce1bc8b011e0d142d34/roles/ead5fac86a0747ea8823a82efcea8ed7 => generated 0 bytes in 38 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 02:02:36.357620 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27a66706-7015-4595-8c4a-f9a0505e8816 admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:02:36.367111 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a709c1cf-dcee-4c73-a4bc-9d2d3c85dd4e None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:02:36.367429 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a709c1cf-dcee-4c73-a4bc-9d2d3c85dd4e None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:02:36.367708 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a709c1cf-dcee-4c73-a4bc-9d2d3c85dd4e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:02:36.399985 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-a709c1cf-dcee-4c73-a4bc-9d2d3c85dd4e None None] MFA Rules not processed for user `d91a28c01d4f4ce1bc8b011e0d142d34`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:02:36.401195 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-a709c1cf-dcee-4c73-a4bc-9d2d3c85dd4e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:02:36.451215 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 908/1835] 10.222.0.27 () {60 vars in 1483 bytes} [Tue Aug 30 02:02:36 2022] PUT /identity/v3/projects/cfde2d9f016e46839128fe2158629c25/users/70f80cd5ef854ad6b7e8f13c18f68864/roles/ead5fac86a0747ea8823a82efcea8ed7 => generated 0 bytes in 137 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 02:02:36.452109 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27a66706-7015-4595-8c4a-f9a0505e8816 admin admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:02:36.516366 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 928/1836] 10.222.0.27 () {58 vars in 958 bytes} [Tue Aug 30 02:02:36 2022] POST /identity/v3/auth/tokens => generated 3329 bytes in 150 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 02:02:36.525276 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a709c1cf-dcee-4c73-a4bc-9d2d3c85dd4e None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:02:36.527157 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-a709c1cf-dcee-4c73-a4bc-9d2d3c85dd4e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:02:36.581396 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27a66706-7015-4595-8c4a-f9a0505e8816 admin admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:02:36.586029 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce681802-5d14-4fdd-85f3-262278399398 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:02:36.586699 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce681802-5d14-4fdd-85f3-262278399398 admin admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:02:36.586979 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce681802-5d14-4fdd-85f3-262278399398 admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:02:36.587381 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce681802-5d14-4fdd-85f3-262278399398 admin admin] PATH_INFO: `/v3/roles` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:02:36.587897 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce681802-5d14-4fdd-85f3-262278399398 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:02:36.589696 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce681802-5d14-4fdd-85f3-262278399398 admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:02:36.606135 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 909/1837] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Aug 30 02:02:36 2022] GET /identity/v3/roles => generated 1420 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 02:02:36.607118 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce681802-5d14-4fdd-85f3-262278399398 admin admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:02:36.619276 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce681802-5d14-4fdd-85f3-262278399398 admin admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:02:36.625125 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ff6b630-f7b6-4ec3-9a8d-dd2d17320f11 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:02:36.633006 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ff6b630-f7b6-4ec3-9a8d-dd2d17320f11 admin admin] REQUEST_METHOD: `PUT` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:02:36.633006 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ff6b630-f7b6-4ec3-9a8d-dd2d17320f11 admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:02:36.633006 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ff6b630-f7b6-4ec3-9a8d-dd2d17320f11 admin admin] PATH_INFO: `/v3/projects/cfde2d9f016e46839128fe2158629c25/users/70f80cd5ef854ad6b7e8f13c18f68864/roles/ead5fac86a0747ea8823a82efcea8ed7` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:02:36.633006 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ff6b630-f7b6-4ec3-9a8d-dd2d17320f11 admin admin] RBAC: Authorizing `identity:create_grant(project_id=cfde2d9f016e46839128fe2158629c25, user_id=70f80cd5ef854ad6b7e8f13c18f68864, role_id=ead5fac86a0747ea8823a82efcea8ed7)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:02:36.633006 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ff6b630-f7b6-4ec3-9a8d-dd2d17320f11 admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:02:36.635531 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a709c1cf-dcee-4c73-a4bc-9d2d3c85dd4e None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:02:36.645012 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 910/1838] 10.222.0.27 () {60 vars in 1483 bytes} [Tue Aug 30 02:02:36 2022] PUT /identity/v3/projects/cfde2d9f016e46839128fe2158629c25/users/70f80cd5ef854ad6b7e8f13c18f68864/roles/ead5fac86a0747ea8823a82efcea8ed7 => generated 0 bytes in 38 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 02:02:36.655623 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bb36e34-31d4-408b-a6ea-9f30f2941476 None None] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:02:36.655994 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bb36e34-31d4-408b-a6ea-9f30f2941476 None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:02:36.656755 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bb36e34-31d4-408b-a6ea-9f30f2941476 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:02:36.737838 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.auth.core [None req-4bb36e34-31d4-408b-a6ea-9f30f2941476 None None] MFA Rules not processed for user `70f80cd5ef854ad6b7e8f13c18f68864`. Rule list: `[]` (Enabled: `True`). {{(pid=81578) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:02:36.737838 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-4bb36e34-31d4-408b-a6ea-9f30f2941476 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:02:36.738071 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2dafcefc-2212-49a8-af44-70dc8e7b4824 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ab936b3767a8417aa80c8c8f1b84b667', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4562f3951fbe465280073e60a11e44e2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:02:36.738071 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dafcefc-2212-49a8-af44-70dc8e7b4824 service cinder] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:02:36.738071 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dafcefc-2212-49a8-af44-70dc8e7b4824 service cinder] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:02:36.738071 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dafcefc-2212-49a8-af44-70dc8e7b4824 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:02:36.738361 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-2dafcefc-2212-49a8-af44-70dc8e7b4824 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:02:36.827465 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2dafcefc-2212-49a8-af44-70dc8e7b4824 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:02:36.831962 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2dafcefc-2212-49a8-af44-70dc8e7b4824 service cinder] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:02:36.859129 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 929/1839] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Aug 30 02:02:36 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 335 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 30 02:02:36.883695 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 911/1840] 10.222.0.27 () {58 vars in 958 bytes} [Tue Aug 30 02:02:36 2022] POST /identity/v3/auth/tokens => generated 3329 bytes in 226 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 02:02:36.892860 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2dafcefc-2212-49a8-af44-70dc8e7b4824 service cinder] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:02:36.894339 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-2dafcefc-2212-49a8-af44-70dc8e7b4824 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:02:36.912313 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2dafcefc-2212-49a8-af44-70dc8e7b4824 service cinder] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:02:37.001975 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1a47f3e-9aeb-4eac-8fb9-14479fab4f18 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ab936b3767a8417aa80c8c8f1b84b667', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4562f3951fbe465280073e60a11e44e2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:02:37.003369 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1a47f3e-9aeb-4eac-8fb9-14479fab4f18 service cinder] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:02:37.003896 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1a47f3e-9aeb-4eac-8fb9-14479fab4f18 service cinder] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:02:37.004377 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1a47f3e-9aeb-4eac-8fb9-14479fab4f18 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:02:37.020143 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1a47f3e-9aeb-4eac-8fb9-14479fab4f18 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:02:37.022723 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1a47f3e-9aeb-4eac-8fb9-14479fab4f18 service cinder] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:02:37.057238 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bb36e34-31d4-408b-a6ea-9f30f2941476 None None] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:02:37.060103 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-4bb36e34-31d4-408b-a6ea-9f30f2941476 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:02:37.065646 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 930/1841] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Aug 30 02:02:36 2022] GET /identity/v3/auth/tokens => generated 3329 bytes in 173 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 02:02:37.132102 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:02:37.153775 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bb36e34-31d4-408b-a6ea-9f30f2941476 None None] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:02:37.158653 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed25b7ec-3a7c-4e1b-808d-7d434b72ff9f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:02:37.159305 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed25b7ec-3a7c-4e1b-808d-7d434b72ff9f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:02:37.159583 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed25b7ec-3a7c-4e1b-808d-7d434b72ff9f admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:02:37.159853 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed25b7ec-3a7c-4e1b-808d-7d434b72ff9f admin admin] PATH_INFO: `/v3/projects/3ce8ab0cd06941d79f67cc35a06bb2e6` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:02:37.160996 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed25b7ec-3a7c-4e1b-808d-7d434b72ff9f admin admin] RBAC: Authorizing `identity:delete_project(project_id=3ce8ab0cd06941d79f67cc35a06bb2e6)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:02:37.162945 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed25b7ec-3a7c-4e1b-808d-7d434b72ff9f admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:02:37.254158 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-ed25b7ec-3a7c-4e1b-808d-7d434b72ff9f admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '3ce8ab0cd06941d79f67cc35a06bb2e6', 'request_id': 'req-ed25b7ec-3a7c-4e1b-808d-7d434b72ff9f'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:02:37.260085 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-ed25b7ec-3a7c-4e1b-808d-7d434b72ff9f admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '3ce8ab0cd06941d79f67cc35a06bb2e6', 'request_id': 'req-ed25b7ec-3a7c-4e1b-808d-7d434b72ff9f'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:02:37.267799 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-ed25b7ec-3a7c-4e1b-808d-7d434b72ff9f admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '3ce8ab0cd06941d79f67cc35a06bb2e6', 'request_id': 'req-ed25b7ec-3a7c-4e1b-808d-7d434b72ff9f'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:02:37.348125 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1a47f3e-9aeb-4eac-8fb9-14479fab4f18 service cinder] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:02:37.351550 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-d1a47f3e-9aeb-4eac-8fb9-14479fab4f18 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:02:37.356461 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-ed25b7ec-3a7c-4e1b-808d-7d434b72ff9f admin admin] The token cache is being invalidate because project 3ce8ab0cd06941d79f67cc35a06bb2e6 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81578) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 30 02:02:37.359610 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-ed25b7ec-3a7c-4e1b-808d-7d434b72ff9f admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:02:37.359610 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-ed25b7ec-3a7c-4e1b-808d-7d434b72ff9f admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:02:37.359610 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 912/1842] 10.222.0.27 () {58 vars in 1233 bytes} [Tue Aug 30 02:02:37 2022] DELETE /identity/v3/projects/3ce8ab0cd06941d79f67cc35a06bb2e6 => generated 0 bytes in 303 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 02:02:37.363256 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed25b7ec-3a7c-4e1b-808d-7d434b72ff9f admin admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:02:37.365003 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-ed25b7ec-3a7c-4e1b-808d-7d434b72ff9f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:02:37.371517 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1a47f3e-9aeb-4eac-8fb9-14479fab4f18 service cinder] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:02:37.435281 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fbdd161-87e6-494c-9909-41f26df4d00d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:02:37.436048 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fbdd161-87e6-494c-9909-41f26df4d00d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:02:37.436328 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fbdd161-87e6-494c-9909-41f26df4d00d admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:02:37.436598 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fbdd161-87e6-494c-9909-41f26df4d00d admin admin] PATH_INFO: `/v3/projects/e0d38af4bc984e459007eccc60f3e117` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:02:37.438073 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fbdd161-87e6-494c-9909-41f26df4d00d admin admin] RBAC: Authorizing `identity:delete_project(project_id=e0d38af4bc984e459007eccc60f3e117)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:02:37.440620 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fbdd161-87e6-494c-9909-41f26df4d00d admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:02:37.489279 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed25b7ec-3a7c-4e1b-808d-7d434b72ff9f admin admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:02:37.494259 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be4c14f5-5a34-4509-9dc8-33ae4b669482 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:02:37.494894 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be4c14f5-5a34-4509-9dc8-33ae4b669482 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:02:37.495175 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be4c14f5-5a34-4509-9dc8-33ae4b669482 admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:02:37.495448 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be4c14f5-5a34-4509-9dc8-33ae4b669482 admin admin] PATH_INFO: `/v3/users/93543dd3fdad44a3b270ee7f7cdc0f86` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:02:37.497321 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be4c14f5-5a34-4509-9dc8-33ae4b669482 admin admin] RBAC: Authorizing `identity:delete_user(user_id=93543dd3fdad44a3b270ee7f7cdc0f86)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:02:37.499322 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be4c14f5-5a34-4509-9dc8-33ae4b669482 admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:02:37.631026 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-7fbdd161-87e6-494c-9909-41f26df4d00d admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'e0d38af4bc984e459007eccc60f3e117', 'request_id': 'req-7fbdd161-87e6-494c-9909-41f26df4d00d'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:02:37.643647 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-7fbdd161-87e6-494c-9909-41f26df4d00d admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'e0d38af4bc984e459007eccc60f3e117', 'request_id': 'req-7fbdd161-87e6-494c-9909-41f26df4d00d'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:02:37.655586 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-7fbdd161-87e6-494c-9909-41f26df4d00d admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'e0d38af4bc984e459007eccc60f3e117', 'request_id': 'req-7fbdd161-87e6-494c-9909-41f26df4d00d'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:02:37.774510 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-7fbdd161-87e6-494c-9909-41f26df4d00d admin admin] The token cache is being invalidate because project e0d38af4bc984e459007eccc60f3e117 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81577) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 30 02:02:37.774858 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-7fbdd161-87e6-494c-9909-41f26df4d00d admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:02:37.775621 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-7fbdd161-87e6-494c-9909-41f26df4d00d admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:02:37.781488 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 931/1843] 10.222.0.27 () {58 vars in 1233 bytes} [Tue Aug 30 02:02:37 2022] DELETE /identity/v3/projects/e0d38af4bc984e459007eccc60f3e117 => generated 0 bytes in 432 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 02:02:37.782778 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fbdd161-87e6-494c-9909-41f26df4d00d admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:02:37.784562 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-7fbdd161-87e6-494c-9909-41f26df4d00d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:02:37.788270 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-be4c14f5-5a34-4509-9dc8-33ae4b669482 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '93543dd3fdad44a3b270ee7f7cdc0f86'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:02:37.789085 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-be4c14f5-5a34-4509-9dc8-33ae4b669482 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '93543dd3fdad44a3b270ee7f7cdc0f86'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:02:37.852378 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fbdd161-87e6-494c-9909-41f26df4d00d admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:02:37.860479 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1fefd99-0706-43b6-b31a-d8cece439e71 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:02:37.860479 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1fefd99-0706-43b6-b31a-d8cece439e71 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:02:37.860479 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1fefd99-0706-43b6-b31a-d8cece439e71 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:02:37.860479 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1fefd99-0706-43b6-b31a-d8cece439e71 admin admin] PATH_INFO: `/v3/users/43136d6e83234361b054b7592f87aaa4` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:02:37.860838 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1fefd99-0706-43b6-b31a-d8cece439e71 admin admin] RBAC: Authorizing `identity:delete_user(user_id=43136d6e83234361b054b7592f87aaa4)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:02:37.862612 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1fefd99-0706-43b6-b31a-d8cece439e71 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:02:37.880219 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-be4c14f5-5a34-4509-9dc8-33ae4b669482 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '93543dd3fdad44a3b270ee7f7cdc0f86'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:02:37.881151 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-be4c14f5-5a34-4509-9dc8-33ae4b669482 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '93543dd3fdad44a3b270ee7f7cdc0f86'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:02:37.889855 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-be4c14f5-5a34-4509-9dc8-33ae4b669482 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '93543dd3fdad44a3b270ee7f7cdc0f86'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:02:37.939126 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 913/1844] 10.222.0.27 () {58 vars in 1224 bytes} [Tue Aug 30 02:02:37 2022] DELETE /identity/v3/users/93543dd3fdad44a3b270ee7f7cdc0f86 => generated 0 bytes in 573 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 30 02:02:38.159182 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-e1fefd99-0706-43b6-b31a-d8cece439e71 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '43136d6e83234361b054b7592f87aaa4'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:02:38.159914 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-e1fefd99-0706-43b6-b31a-d8cece439e71 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '43136d6e83234361b054b7592f87aaa4'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:02:38.160627 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-e1fefd99-0706-43b6-b31a-d8cece439e71 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '43136d6e83234361b054b7592f87aaa4'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:02:38.174922 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-e1fefd99-0706-43b6-b31a-d8cece439e71 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '43136d6e83234361b054b7592f87aaa4'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:02:38.184236 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-e1fefd99-0706-43b6-b31a-d8cece439e71 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '43136d6e83234361b054b7592f87aaa4'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:02:38.275970 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 932/1845] 10.222.0.27 () {58 vars in 1224 bytes} [Tue Aug 30 02:02:37 2022] DELETE /identity/v3/users/43136d6e83234361b054b7592f87aaa4 => generated 0 bytes in 494 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 02:02:38.642625 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be4c14f5-5a34-4509-9dc8-33ae4b669482 admin admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:02:38.649891 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1fefd99-0706-43b6-b31a-d8cece439e71 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:02:38.650427 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-be4c14f5-5a34-4509-9dc8-33ae4b669482 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:02:38.653562 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-e1fefd99-0706-43b6-b31a-d8cece439e71 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:02:38.790550 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1fefd99-0706-43b6-b31a-d8cece439e71 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:02:38.797767 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ad60ae8-eb40-4bf6-a8f1-697df77af3bb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:02:38.798722 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ad60ae8-eb40-4bf6-a8f1-697df77af3bb admin admin] REQUEST_METHOD: `DELETE` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:02:38.799165 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ad60ae8-eb40-4bf6-a8f1-697df77af3bb admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:02:38.799604 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ad60ae8-eb40-4bf6-a8f1-697df77af3bb admin admin] PATH_INFO: `/v3/users/111b7de24c9a4a079b79b9da5c66bb64` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:02:38.801104 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ad60ae8-eb40-4bf6-a8f1-697df77af3bb admin admin] RBAC: Authorizing `identity:delete_user(user_id=111b7de24c9a4a079b79b9da5c66bb64)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:02:38.801860 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=82,keystone:DELETE=16,keystone:INSERT=2 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:02:38.803235 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ad60ae8-eb40-4bf6-a8f1-697df77af3bb admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:02:38.866038 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be4c14f5-5a34-4509-9dc8-33ae4b669482 admin admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:02:38.889091 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69d700b2-7a9e-4bf4-92ab-0be1e45589c8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:02:38.889091 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69d700b2-7a9e-4bf4-92ab-0be1e45589c8 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:02:38.889091 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69d700b2-7a9e-4bf4-92ab-0be1e45589c8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:02:38.889091 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69d700b2-7a9e-4bf4-92ab-0be1e45589c8 admin admin] PATH_INFO: `/v3/projects/a0d103ed4e9246e29c86e461cadbdee6` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:02:38.890748 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69d700b2-7a9e-4bf4-92ab-0be1e45589c8 admin admin] RBAC: Authorizing `identity:delete_project(project_id=a0d103ed4e9246e29c86e461cadbdee6)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:02:38.894351 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69d700b2-7a9e-4bf4-92ab-0be1e45589c8 admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:02:39.052024 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-6ad60ae8-eb40-4bf6-a8f1-697df77af3bb admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '111b7de24c9a4a079b79b9da5c66bb64'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:02:39.053186 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-6ad60ae8-eb40-4bf6-a8f1-697df77af3bb admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '111b7de24c9a4a079b79b9da5c66bb64'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:02:39.055331 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-6ad60ae8-eb40-4bf6-a8f1-697df77af3bb admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '111b7de24c9a4a079b79b9da5c66bb64'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:02:39.080626 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-69d700b2-7a9e-4bf4-92ab-0be1e45589c8 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'a0d103ed4e9246e29c86e461cadbdee6', 'request_id': 'req-69d700b2-7a9e-4bf4-92ab-0be1e45589c8'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:02:39.091449 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-6ad60ae8-eb40-4bf6-a8f1-697df77af3bb admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '111b7de24c9a4a079b79b9da5c66bb64'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:02:39.102130 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-6ad60ae8-eb40-4bf6-a8f1-697df77af3bb admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '111b7de24c9a4a079b79b9da5c66bb64'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:02:39.117259 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-69d700b2-7a9e-4bf4-92ab-0be1e45589c8 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'a0d103ed4e9246e29c86e461cadbdee6', 'request_id': 'req-69d700b2-7a9e-4bf4-92ab-0be1e45589c8'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:02:39.160010 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-69d700b2-7a9e-4bf4-92ab-0be1e45589c8 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'a0d103ed4e9246e29c86e461cadbdee6', 'request_id': 'req-69d700b2-7a9e-4bf4-92ab-0be1e45589c8'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:02:39.237414 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 933/1846] 10.222.0.27 () {58 vars in 1224 bytes} [Tue Aug 30 02:02:38 2022] DELETE /identity/v3/users/111b7de24c9a4a079b79b9da5c66bb64 => generated 0 bytes in 589 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 30 02:02:39.238805 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ad60ae8-eb40-4bf6-a8f1-697df77af3bb admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:02:39.240744 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-6ad60ae8-eb40-4bf6-a8f1-697df77af3bb admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:02:39.333235 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ad60ae8-eb40-4bf6-a8f1-697df77af3bb admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:02:39.337596 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a945a6fc-022e-428e-afeb-8bbe6fc772d9 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ab936b3767a8417aa80c8c8f1b84b667', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4562f3951fbe465280073e60a11e44e2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:02:39.338261 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a945a6fc-022e-428e-afeb-8bbe6fc772d9 service cinder] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:02:39.340049 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a945a6fc-022e-428e-afeb-8bbe6fc772d9 service cinder] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:02:39.340049 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a945a6fc-022e-428e-afeb-8bbe6fc772d9 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:02:39.342349 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-a945a6fc-022e-428e-afeb-8bbe6fc772d9 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:02:39.434581 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a945a6fc-022e-428e-afeb-8bbe6fc772d9 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:02:39.436047 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a945a6fc-022e-428e-afeb-8bbe6fc772d9 service cinder] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:02:39.451015 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 934/1847] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Aug 30 02:02:39 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 213 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 02:02:39.453185 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b206fa3-8f63-40a2-81a5-ba916cdfa5e3 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:02:39.453689 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b206fa3-8f63-40a2-81a5-ba916cdfa5e3 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:02:39.454122 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b206fa3-8f63-40a2-81a5-ba916cdfa5e3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:02:39.475710 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-69d700b2-7a9e-4bf4-92ab-0be1e45589c8 admin admin] The token cache is being invalidate because project a0d103ed4e9246e29c86e461cadbdee6 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81578) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 30 02:02:39.478909 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-69d700b2-7a9e-4bf4-92ab-0be1e45589c8 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:02:39.478909 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-69d700b2-7a9e-4bf4-92ab-0be1e45589c8 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:02:39.479111 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-5b206fa3-8f63-40a2-81a5-ba916cdfa5e3 None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:02:39.479795 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-5b206fa3-8f63-40a2-81a5-ba916cdfa5e3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:02:39.489268 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 914/1848] 10.222.0.27 () {58 vars in 1233 bytes} [Tue Aug 30 02:02:38 2022] DELETE /identity/v3/projects/a0d103ed4e9246e29c86e461cadbdee6 => generated 0 bytes in 844 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 02:02:39.490991 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc5c9f4b-0191-400f-b199-813f993e836d None None] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:02:39.491347 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc5c9f4b-0191-400f-b199-813f993e836d None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:02:39.491691 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc5c9f4b-0191-400f-b199-813f993e836d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:02:39.518860 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.auth.core [None req-fc5c9f4b-0191-400f-b199-813f993e836d None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81578) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:02:39.520345 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-fc5c9f4b-0191-400f-b199-813f993e836d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:02:39.587574 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 935/1849] 10.222.0.27 () {58 vars in 958 bytes} [Tue Aug 30 02:02:39 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 136 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 02:02:39.588736 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b206fa3-8f63-40a2-81a5-ba916cdfa5e3 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:02:39.592383 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-5b206fa3-8f63-40a2-81a5-ba916cdfa5e3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:02:39.611611 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b206fa3-8f63-40a2-81a5-ba916cdfa5e3 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:02:39.630276 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb4900ca-23d4-4a55-a108-b7e30bbd93fc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:02:39.633537 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb4900ca-23d4-4a55-a108-b7e30bbd93fc admin admin] REQUEST_METHOD: `DELETE` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:02:39.633537 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb4900ca-23d4-4a55-a108-b7e30bbd93fc admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:02:39.633537 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb4900ca-23d4-4a55-a108-b7e30bbd93fc admin admin] PATH_INFO: `/v3/projects/264f8a2690f64ce6abcf196780b73b9b` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:02:39.639015 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb4900ca-23d4-4a55-a108-b7e30bbd93fc admin admin] RBAC: Authorizing `identity:delete_project(project_id=264f8a2690f64ce6abcf196780b73b9b)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:02:39.655992 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb4900ca-23d4-4a55-a108-b7e30bbd93fc admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:02:39.668042 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 915/1850] 10.222.0.27 () {58 vars in 958 bytes} [Tue Aug 30 02:02:39 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 172 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 02:02:39.668042 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc5c9f4b-0191-400f-b199-813f993e836d None None] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:02:39.670067 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-fc5c9f4b-0191-400f-b199-813f993e836d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:02:39.755743 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc5c9f4b-0191-400f-b199-813f993e836d None None] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:02:39.756568 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d1efa1a-40b9-4a3f-9e62-cf2225d492d5 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f3ea1580c56f4814a92bbb2acf542905', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4562f3951fbe465280073e60a11e44e2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:02:39.757437 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d1efa1a-40b9-4a3f-9e62-cf2225d492d5 service neutron] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:02:39.757905 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d1efa1a-40b9-4a3f-9e62-cf2225d492d5 service neutron] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:02:39.758323 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d1efa1a-40b9-4a3f-9e62-cf2225d492d5 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:02:39.779266 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d1efa1a-40b9-4a3f-9e62-cf2225d492d5 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:02:39.780705 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d1efa1a-40b9-4a3f-9e62-cf2225d492d5 service neutron] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:02:39.791572 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 916/1851] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Aug 30 02:02:39 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 129 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 02:02:39.793052 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-384b7417-98e8-43ed-9052-3f0b803ae547 None None] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:02:39.793553 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-384b7417-98e8-43ed-9052-3f0b803ae547 None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:02:39.794110 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-384b7417-98e8-43ed-9052-3f0b803ae547 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:02:39.820332 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.auth.core [None req-384b7417-98e8-43ed-9052-3f0b803ae547 None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81578) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:02:39.822233 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-384b7417-98e8-43ed-9052-3f0b803ae547 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:02:39.834908 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-bb4900ca-23d4-4a55-a108-b7e30bbd93fc admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '264f8a2690f64ce6abcf196780b73b9b', 'request_id': 'req-bb4900ca-23d4-4a55-a108-b7e30bbd93fc'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:02:39.850815 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-bb4900ca-23d4-4a55-a108-b7e30bbd93fc admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '264f8a2690f64ce6abcf196780b73b9b', 'request_id': 'req-bb4900ca-23d4-4a55-a108-b7e30bbd93fc'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:02:39.859261 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-bb4900ca-23d4-4a55-a108-b7e30bbd93fc admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '264f8a2690f64ce6abcf196780b73b9b', 'request_id': 'req-bb4900ca-23d4-4a55-a108-b7e30bbd93fc'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:02:39.910769 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 917/1852] 10.222.0.27 () {58 vars in 958 bytes} [Tue Aug 30 02:02:39 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 119 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 02:02:39.914807 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-384b7417-98e8-43ed-9052-3f0b803ae547 None None] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:02:39.925776 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-384b7417-98e8-43ed-9052-3f0b803ae547 None None] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:02:39.931581 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2e49ecb-961f-431f-9cdf-7337f1b9667a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:02:39.932207 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2e49ecb-961f-431f-9cdf-7337f1b9667a admin admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:02:39.932488 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2e49ecb-961f-431f-9cdf-7337f1b9667a admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:02:39.932759 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2e49ecb-961f-431f-9cdf-7337f1b9667a admin admin] PATH_INFO: `/v3/domains` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:02:39.934242 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2e49ecb-961f-431f-9cdf-7337f1b9667a admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81578) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 02:02:39.934583 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2e49ecb-961f-431f-9cdf-7337f1b9667a admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:02:39.936228 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2e49ecb-961f-431f-9cdf-7337f1b9667a admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:02:39.954248 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 918/1853] 10.222.0.27 () {58 vars in 1153 bytes} [Tue Aug 30 02:02:39 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 40 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 30 02:02:39.959223 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2e49ecb-961f-431f-9cdf-7337f1b9667a admin admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:02:39.971099 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2e49ecb-961f-431f-9cdf-7337f1b9667a admin admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:02:39.989320 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ebc887e-6c95-4114-8b53-0a158383e208 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:02:39.989320 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ebc887e-6c95-4114-8b53-0a158383e208 admin admin] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:02:39.989320 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ebc887e-6c95-4114-8b53-0a158383e208 admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:02:39.989820 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ebc887e-6c95-4114-8b53-0a158383e208 admin admin] PATH_INFO: `/v3/projects` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:02:39.990540 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ebc887e-6c95-4114-8b53-0a158383e208 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:02:39.993792 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ebc887e-6c95-4114-8b53-0a158383e208 admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:02:40.026130 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-bb4900ca-23d4-4a55-a108-b7e30bbd93fc admin admin] The token cache is being invalidate because project 264f8a2690f64ce6abcf196780b73b9b was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81577) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 30 02:02:40.031015 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-bb4900ca-23d4-4a55-a108-b7e30bbd93fc admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:02:40.031015 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-bb4900ca-23d4-4a55-a108-b7e30bbd93fc admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:02:40.031015 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 936/1854] 10.222.0.27 () {58 vars in 1233 bytes} [Tue Aug 30 02:02:39 2022] DELETE /identity/v3/projects/264f8a2690f64ce6abcf196780b73b9b => generated 0 bytes in 441 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 02:02:40.034885 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05bafcf2-3ea8-4eae-93f0-c063c466886d None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:02:40.035246 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05bafcf2-3ea8-4eae-93f0-c063c466886d None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:02:40.035642 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05bafcf2-3ea8-4eae-93f0-c063c466886d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:02:40.058879 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-05bafcf2-3ea8-4eae-93f0-c063c466886d None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:02:40.060434 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-05bafcf2-3ea8-4eae-93f0-c063c466886d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:02:40.072173 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 937/1855] 10.222.0.27 () {58 vars in 958 bytes} [Tue Aug 30 02:02:40 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 38 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 02:02:40.079852 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fcecd63-0ba0-44fc-bf70-de7638832f70 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:02:40.080146 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fcecd63-0ba0-44fc-bf70-de7638832f70 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:02:40.080486 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fcecd63-0ba0-44fc-bf70-de7638832f70 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:02:40.104029 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-5fcecd63-0ba0-44fc-bf70-de7638832f70 None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:02:40.105830 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-5fcecd63-0ba0-44fc-bf70-de7638832f70 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:02:40.125374 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 938/1856] 10.222.0.27 () {58 vars in 958 bytes} [Tue Aug 30 02:02:40 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 40 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 02:02:40.125374 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fcecd63-0ba0-44fc-bf70-de7638832f70 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:02:40.138877 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fcecd63-0ba0-44fc-bf70-de7638832f70 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:02:40.144694 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d83089aa-1107-4890-b74b-2da699b30945 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:02:40.145360 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d83089aa-1107-4890-b74b-2da699b30945 admin admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:02:40.145712 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d83089aa-1107-4890-b74b-2da699b30945 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:02:40.146092 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d83089aa-1107-4890-b74b-2da699b30945 admin admin] PATH_INFO: `/v3/domains` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:02:40.147050 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d83089aa-1107-4890-b74b-2da699b30945 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 02:02:40.147366 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d83089aa-1107-4890-b74b-2da699b30945 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:02:40.149057 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d83089aa-1107-4890-b74b-2da699b30945 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:02:40.150875 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 919/1857] 10.222.0.27 () {60 vars in 1153 bytes} [Tue Aug 30 02:02:39 2022] POST /identity/v3/projects => generated 363 bytes in 193 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 02:02:40.156499 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ebc887e-6c95-4114-8b53-0a158383e208 admin admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:02:40.158453 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-2ebc887e-6c95-4114-8b53-0a158383e208 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:02:40.174991 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 939/1858] 10.222.0.27 () {58 vars in 1153 bytes} [Tue Aug 30 02:02:40 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 56 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 02:02:40.176187 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d83089aa-1107-4890-b74b-2da699b30945 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:02:40.196801 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d83089aa-1107-4890-b74b-2da699b30945 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:02:40.276357 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60fdcf8b-ae37-43f8-b363-792690c4cb33 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:02:40.277708 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60fdcf8b-ae37-43f8-b363-792690c4cb33 admin admin] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:02:40.278091 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60fdcf8b-ae37-43f8-b363-792690c4cb33 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:02:40.278444 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60fdcf8b-ae37-43f8-b363-792690c4cb33 admin admin] PATH_INFO: `/v3/projects` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:02:40.279672 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60fdcf8b-ae37-43f8-b363-792690c4cb33 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:02:40.282361 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60fdcf8b-ae37-43f8-b363-792690c4cb33 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:02:40.288257 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ebc887e-6c95-4114-8b53-0a158383e208 admin admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:02:40.295331 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1aeb7e2-3809-4436-8213-4e001e6982c9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:02:40.297611 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1aeb7e2-3809-4436-8213-4e001e6982c9 admin admin] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:02:40.298225 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1aeb7e2-3809-4436-8213-4e001e6982c9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:02:40.298728 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1aeb7e2-3809-4436-8213-4e001e6982c9 admin admin] PATH_INFO: `/v3/users` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:02:40.299583 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1aeb7e2-3809-4436-8213-4e001e6982c9 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:02:40.301974 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1aeb7e2-3809-4436-8213-4e001e6982c9 admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:02:40.439180 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 940/1859] 10.222.0.27 () {60 vars in 1153 bytes} [Tue Aug 30 02:02:40 2022] POST /identity/v3/projects => generated 367 bytes in 257 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 02:02:40.439180 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60fdcf8b-ae37-43f8-b363-792690c4cb33 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:02:40.577272 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 920/1860] 10.222.0.27 () {60 vars in 1144 bytes} [Tue Aug 30 02:02:40 2022] POST /identity/v3/users => generated 282 bytes in 333 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 02:02:40.577272 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1aeb7e2-3809-4436-8213-4e001e6982c9 admin admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:02:40.577272 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1aeb7e2-3809-4436-8213-4e001e6982c9 admin admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:02:40.614971 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60fdcf8b-ae37-43f8-b363-792690c4cb33 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:02:40.629906 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e24413e-2c22-45d2-a654-ad9a377a8aa1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:02:40.630547 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e24413e-2c22-45d2-a654-ad9a377a8aa1 admin admin] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:02:40.631717 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e24413e-2c22-45d2-a654-ad9a377a8aa1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:02:40.631993 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e24413e-2c22-45d2-a654-ad9a377a8aa1 admin admin] PATH_INFO: `/v3/users` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:02:40.633272 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e24413e-2c22-45d2-a654-ad9a377a8aa1 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:02:40.638955 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e24413e-2c22-45d2-a654-ad9a377a8aa1 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:02:40.688071 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d7c6a6e-09c3-4a81-9818-0f6eb25f2e70 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:02:40.688607 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d7c6a6e-09c3-4a81-9818-0f6eb25f2e70 admin admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:02:40.688607 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d7c6a6e-09c3-4a81-9818-0f6eb25f2e70 admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:02:40.688835 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d7c6a6e-09c3-4a81-9818-0f6eb25f2e70 admin admin] PATH_INFO: `/v3/roles` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:02:40.691447 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d7c6a6e-09c3-4a81-9818-0f6eb25f2e70 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:02:40.694005 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d7c6a6e-09c3-4a81-9818-0f6eb25f2e70 admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:02:40.710807 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 921/1861] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Aug 30 02:02:40 2022] GET /identity/v3/roles => generated 1420 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 30 02:02:40.712008 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d7c6a6e-09c3-4a81-9818-0f6eb25f2e70 admin admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:02:40.730851 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-3d7c6a6e-09c3-4a81-9818-0f6eb25f2e70 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:02:40.750821 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d7c6a6e-09c3-4a81-9818-0f6eb25f2e70 admin admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:02:40.758007 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5512bc0-2ff8-406a-88b2-be6fdc332e97 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:02:40.758658 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5512bc0-2ff8-406a-88b2-be6fdc332e97 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:02:40.759078 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5512bc0-2ff8-406a-88b2-be6fdc332e97 admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:02:40.759354 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5512bc0-2ff8-406a-88b2-be6fdc332e97 admin admin] PATH_INFO: `/v3/projects/bc6021ca1ab14ac684ad7b6d8fab1efa` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:02:40.760592 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5512bc0-2ff8-406a-88b2-be6fdc332e97 admin admin] RBAC: Authorizing `identity:delete_project(project_id=bc6021ca1ab14ac684ad7b6d8fab1efa)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:02:40.762945 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5512bc0-2ff8-406a-88b2-be6fdc332e97 admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:02:40.773200 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 941/1862] 10.222.0.27 () {60 vars in 1144 bytes} [Tue Aug 30 02:02:40 2022] POST /identity/v3/users => generated 284 bytes in 336 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 02:02:40.774907 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e24413e-2c22-45d2-a654-ad9a377a8aa1 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:02:40.799264 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e24413e-2c22-45d2-a654-ad9a377a8aa1 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:02:40.802361 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:02:40.818258 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf57b63a-c6eb-406a-a34c-1ce5993af301 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:02:40.819154 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf57b63a-c6eb-406a-a34c-1ce5993af301 admin admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:02:40.819629 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf57b63a-c6eb-406a-a34c-1ce5993af301 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:02:40.820048 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf57b63a-c6eb-406a-a34c-1ce5993af301 admin admin] PATH_INFO: `/v3/roles` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:02:40.820891 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf57b63a-c6eb-406a-a34c-1ce5993af301 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:02:40.822835 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf57b63a-c6eb-406a-a34c-1ce5993af301 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:02:40.858927 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 942/1863] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Aug 30 02:02:40 2022] GET /identity/v3/roles => generated 1420 bytes in 85 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 02:02:40.860111 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf57b63a-c6eb-406a-a34c-1ce5993af301 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:02:40.882626 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf57b63a-c6eb-406a-a34c-1ce5993af301 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:02:40.888556 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba8d4ce3-7d28-44fb-8be1-45e511e2b309 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:02:40.889188 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba8d4ce3-7d28-44fb-8be1-45e511e2b309 admin admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:02:40.889545 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba8d4ce3-7d28-44fb-8be1-45e511e2b309 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:02:40.889780 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba8d4ce3-7d28-44fb-8be1-45e511e2b309 admin admin] PATH_INFO: `/v3/roles` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:02:40.891264 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba8d4ce3-7d28-44fb-8be1-45e511e2b309 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:02:40.894263 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba8d4ce3-7d28-44fb-8be1-45e511e2b309 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:02:40.909510 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 943/1864] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Aug 30 02:02:40 2022] GET /identity/v3/roles => generated 1420 bytes in 50 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 02:02:40.910913 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba8d4ce3-7d28-44fb-8be1-45e511e2b309 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:02:40.927847 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba8d4ce3-7d28-44fb-8be1-45e511e2b309 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:02:40.943429 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-a5512bc0-2ff8-406a-88b2-be6fdc332e97 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'bc6021ca1ab14ac684ad7b6d8fab1efa', 'request_id': 'req-a5512bc0-2ff8-406a-88b2-be6fdc332e97'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:02:40.954521 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-a5512bc0-2ff8-406a-88b2-be6fdc332e97 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'bc6021ca1ab14ac684ad7b6d8fab1efa', 'request_id': 'req-a5512bc0-2ff8-406a-88b2-be6fdc332e97'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:02:40.966482 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-a5512bc0-2ff8-406a-88b2-be6fdc332e97 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'bc6021ca1ab14ac684ad7b6d8fab1efa', 'request_id': 'req-a5512bc0-2ff8-406a-88b2-be6fdc332e97'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:02:41.044682 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eca97019-03d9-4332-bd27-6a76e34ad41c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:02:41.045539 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eca97019-03d9-4332-bd27-6a76e34ad41c admin admin] REQUEST_METHOD: `PUT` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:02:41.045904 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eca97019-03d9-4332-bd27-6a76e34ad41c admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:02:41.046157 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eca97019-03d9-4332-bd27-6a76e34ad41c admin admin] PATH_INFO: `/v3/projects/dcef17b6bb014685adcc154bb43c8ec8/users/4e8bff45c40f4848b3cd4f32c5ad0871/roles/1fb81afcb9f245f3b989add94598a063` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:02:41.047202 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-a5512bc0-2ff8-406a-88b2-be6fdc332e97 admin admin] The token cache is being invalidate because project bc6021ca1ab14ac684ad7b6d8fab1efa was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81578) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 30 02:02:41.047552 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-a5512bc0-2ff8-406a-88b2-be6fdc332e97 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:02:41.048227 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-a5512bc0-2ff8-406a-88b2-be6fdc332e97 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:02:41.050434 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 922/1865] 10.222.0.27 () {58 vars in 1233 bytes} [Tue Aug 30 02:02:40 2022] DELETE /identity/v3/projects/bc6021ca1ab14ac684ad7b6d8fab1efa => generated 0 bytes in 339 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 02:02:41.051266 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5512bc0-2ff8-406a-88b2-be6fdc332e97 admin admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:02:41.054118 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-a5512bc0-2ff8-406a-88b2-be6fdc332e97 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:02:41.066203 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eca97019-03d9-4332-bd27-6a76e34ad41c admin admin] RBAC: Authorizing `identity:create_grant(project_id=dcef17b6bb014685adcc154bb43c8ec8, user_id=4e8bff45c40f4848b3cd4f32c5ad0871, role_id=1fb81afcb9f245f3b989add94598a063)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:02:41.069965 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5512bc0-2ff8-406a-88b2-be6fdc332e97 admin admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:02:41.075114 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eca97019-03d9-4332-bd27-6a76e34ad41c admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:02:41.082121 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e44de87-2618-406a-9c69-7342178afcab admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:02:41.083567 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e44de87-2618-406a-9c69-7342178afcab admin admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:02:41.083937 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e44de87-2618-406a-9c69-7342178afcab admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:02:41.084284 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e44de87-2618-406a-9c69-7342178afcab admin admin] PATH_INFO: `/v3/roles` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:02:41.085042 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e44de87-2618-406a-9c69-7342178afcab admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:02:41.086691 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e44de87-2618-406a-9c69-7342178afcab admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:02:41.100777 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 923/1866] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Aug 30 02:02:41 2022] GET /identity/v3/roles => generated 1420 bytes in 50 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 02:02:41.102398 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e44de87-2618-406a-9c69-7342178afcab admin admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:02:41.105097 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-0e44de87-2618-406a-9c69-7342178afcab admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:02:41.123101 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e44de87-2618-406a-9c69-7342178afcab admin admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:02:41.130354 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b60d8984-92a3-4f53-b237-815a74f69544 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:02:41.141582 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b60d8984-92a3-4f53-b237-815a74f69544 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:02:41.141582 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b60d8984-92a3-4f53-b237-815a74f69544 admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:02:41.141582 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b60d8984-92a3-4f53-b237-815a74f69544 admin admin] PATH_INFO: `/v3/users/70f80cd5ef854ad6b7e8f13c18f68864` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:02:41.141582 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b60d8984-92a3-4f53-b237-815a74f69544 admin admin] RBAC: Authorizing `identity:delete_user(user_id=70f80cd5ef854ad6b7e8f13c18f68864)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:02:41.141582 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b60d8984-92a3-4f53-b237-815a74f69544 admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:02:41.266197 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 944/1867] 10.222.0.27 () {60 vars in 1483 bytes} [Tue Aug 30 02:02:40 2022] PUT /identity/v3/projects/dcef17b6bb014685adcc154bb43c8ec8/users/4e8bff45c40f4848b3cd4f32c5ad0871/roles/1fb81afcb9f245f3b989add94598a063 => generated 0 bytes in 356 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 02:02:41.278747 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eca97019-03d9-4332-bd27-6a76e34ad41c admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:02:41.353637 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eca97019-03d9-4332-bd27-6a76e34ad41c admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:02:41.358946 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40445714-908a-4d09-8faf-b1d15d91e143 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:02:41.359612 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40445714-908a-4d09-8faf-b1d15d91e143 admin admin] REQUEST_METHOD: `PUT` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:02:41.359890 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40445714-908a-4d09-8faf-b1d15d91e143 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:02:41.360161 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40445714-908a-4d09-8faf-b1d15d91e143 admin admin] PATH_INFO: `/v3/projects/bb6da553928244dfb44d34212725fce6/users/e9101336c65047fea32ed1212dddb547/roles/1fb81afcb9f245f3b989add94598a063` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:02:41.375933 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40445714-908a-4d09-8faf-b1d15d91e143 admin admin] RBAC: Authorizing `identity:create_grant(project_id=bb6da553928244dfb44d34212725fce6, user_id=e9101336c65047fea32ed1212dddb547, role_id=1fb81afcb9f245f3b989add94598a063)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:02:41.380577 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40445714-908a-4d09-8faf-b1d15d91e143 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:02:41.490602 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-b60d8984-92a3-4f53-b237-815a74f69544 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '70f80cd5ef854ad6b7e8f13c18f68864'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:02:41.491309 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-b60d8984-92a3-4f53-b237-815a74f69544 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '70f80cd5ef854ad6b7e8f13c18f68864'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:02:41.537182 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 945/1868] 10.222.0.27 () {60 vars in 1483 bytes} [Tue Aug 30 02:02:41 2022] PUT /identity/v3/projects/bb6da553928244dfb44d34212725fce6/users/e9101336c65047fea32ed1212dddb547/roles/1fb81afcb9f245f3b989add94598a063 => generated 0 bytes in 270 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 02:02:41.538746 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d037bbd-636e-44fd-965f-21cd3ef9db72 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:02:41.539033 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d037bbd-636e-44fd-965f-21cd3ef9db72 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:02:41.539306 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d037bbd-636e-44fd-965f-21cd3ef9db72 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:02:41.555060 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-b60d8984-92a3-4f53-b237-815a74f69544 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '70f80cd5ef854ad6b7e8f13c18f68864'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:02:41.555885 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-b60d8984-92a3-4f53-b237-815a74f69544 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '70f80cd5ef854ad6b7e8f13c18f68864'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:02:41.569879 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-b60d8984-92a3-4f53-b237-815a74f69544 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '70f80cd5ef854ad6b7e8f13c18f68864'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:02:41.575905 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-1d037bbd-636e-44fd-965f-21cd3ef9db72 None None] MFA Rules not processed for user `4e8bff45c40f4848b3cd4f32c5ad0871`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:02:41.579217 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-1d037bbd-636e-44fd-965f-21cd3ef9db72 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:02:41.603881 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 924/1869] 10.222.0.27 () {58 vars in 1224 bytes} [Tue Aug 30 02:02:41 2022] DELETE /identity/v3/users/70f80cd5ef854ad6b7e8f13c18f68864 => generated 0 bytes in 502 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 02:02:41.606015 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce438097-837d-4083-8e83-d3bca7a3fec6 None None] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:02:41.606319 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce438097-837d-4083-8e83-d3bca7a3fec6 None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:02:41.606687 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce438097-837d-4083-8e83-d3bca7a3fec6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:02:41.661316 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.auth.core [None req-ce438097-837d-4083-8e83-d3bca7a3fec6 None None] MFA Rules not processed for user `e9101336c65047fea32ed1212dddb547`. Rule list: `[]` (Enabled: `True`). {{(pid=81578) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:02:41.664337 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-ce438097-837d-4083-8e83-d3bca7a3fec6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:02:41.696597 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 946/1870] 10.222.0.27 () {58 vars in 958 bytes} [Tue Aug 30 02:02:41 2022] POST /identity/v3/auth/tokens => generated 3240 bytes in 158 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 02:02:41.698104 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d037bbd-636e-44fd-965f-21cd3ef9db72 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:02:41.700835 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-1d037bbd-636e-44fd-965f-21cd3ef9db72 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:02:41.751688 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 925/1871] 10.222.0.27 () {58 vars in 958 bytes} [Tue Aug 30 02:02:41 2022] POST /identity/v3/auth/tokens => generated 3244 bytes in 147 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 02:02:41.755981 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce438097-837d-4083-8e83-d3bca7a3fec6 None None] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:02:41.764306 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-ce438097-837d-4083-8e83-d3bca7a3fec6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:02:41.841517 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d037bbd-636e-44fd-965f-21cd3ef9db72 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:02:41.922226 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1501fc3-e012-4ce4-9c1c-d295b709cc2a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:02:41.930781 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce438097-837d-4083-8e83-d3bca7a3fec6 None None] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:02:41.935601 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1501fc3-e012-4ce4-9c1c-d295b709cc2a admin admin] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:02:41.935895 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1501fc3-e012-4ce4-9c1c-d295b709cc2a admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:02:41.936374 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1501fc3-e012-4ce4-9c1c-d295b709cc2a admin admin] PATH_INFO: `/v3/projects` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:02:41.936877 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1501fc3-e012-4ce4-9c1c-d295b709cc2a admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:02:41.943302 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1501fc3-e012-4ce4-9c1c-d295b709cc2a admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:02:41.946826 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b92026cc-becc-4b4f-aa93-0bee32a7a527 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:02:41.947564 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b92026cc-becc-4b4f-aa93-0bee32a7a527 admin admin] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:02:41.947942 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b92026cc-becc-4b4f-aa93-0bee32a7a527 admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:02:41.948394 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b92026cc-becc-4b4f-aa93-0bee32a7a527 admin admin] PATH_INFO: `/v3/projects` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:02:41.949376 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b92026cc-becc-4b4f-aa93-0bee32a7a527 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:02:41.954279 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b92026cc-becc-4b4f-aa93-0bee32a7a527 admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:02:42.129312 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 947/1873] 10.222.0.27 () {60 vars in 1153 bytes} [Tue Aug 30 02:02:41 2022] POST /identity/v3/projects => generated 363 bytes in 432 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 02:02:42.129547 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 926/1873] 10.222.0.27 () {60 vars in 1153 bytes} [Tue Aug 30 02:02:41 2022] POST /identity/v3/projects => generated 369 bytes in 374 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 30 02:02:42.136111 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b92026cc-becc-4b4f-aa93-0bee32a7a527 admin admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:02:42.136290 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1501fc3-e012-4ce4-9c1c-d295b709cc2a admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:02:42.136392 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-b92026cc-becc-4b4f-aa93-0bee32a7a527 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:02:42.237156 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1501fc3-e012-4ce4-9c1c-d295b709cc2a admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:02:42.238428 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b92026cc-becc-4b4f-aa93-0bee32a7a527 admin admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:02:42.248970 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efc55742-be28-4c73-9e84-4d8710223645 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:02:42.250193 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efc55742-be28-4c73-9e84-4d8710223645 admin admin] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:02:42.250483 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efc55742-be28-4c73-9e84-4d8710223645 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:02:42.250768 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efc55742-be28-4c73-9e84-4d8710223645 admin admin] PATH_INFO: `/v3/users` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:02:42.251543 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efc55742-be28-4c73-9e84-4d8710223645 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:02:42.253398 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efc55742-be28-4c73-9e84-4d8710223645 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:02:42.332464 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b699cf9f-3ddb-47ce-8920-92cfe99aea61 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ab936b3767a8417aa80c8c8f1b84b667', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4562f3951fbe465280073e60a11e44e2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:02:42.333078 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b699cf9f-3ddb-47ce-8920-92cfe99aea61 service cinder] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:02:42.339854 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b699cf9f-3ddb-47ce-8920-92cfe99aea61 service cinder] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:02:42.339854 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b699cf9f-3ddb-47ce-8920-92cfe99aea61 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:02:42.339854 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-b699cf9f-3ddb-47ce-8920-92cfe99aea61 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:02:42.396747 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 948/1874] 10.222.0.27 () {60 vars in 1144 bytes} [Tue Aug 30 02:02:42 2022] POST /identity/v3/users => generated 288 bytes in 264 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 02:02:42.401012 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efc55742-be28-4c73-9e84-4d8710223645 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:02:42.415260 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efc55742-be28-4c73-9e84-4d8710223645 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:02:42.427675 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1eeb4cb4-c6a7-4fb2-95fa-1ad4e04e904b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:02:42.428984 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1eeb4cb4-c6a7-4fb2-95fa-1ad4e04e904b admin admin] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:02:42.429256 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1eeb4cb4-c6a7-4fb2-95fa-1ad4e04e904b admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:02:42.429620 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1eeb4cb4-c6a7-4fb2-95fa-1ad4e04e904b admin admin] PATH_INFO: `/v3/users` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:02:42.430396 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1eeb4cb4-c6a7-4fb2-95fa-1ad4e04e904b admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:02:42.432413 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1eeb4cb4-c6a7-4fb2-95fa-1ad4e04e904b admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:02:42.440226 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b699cf9f-3ddb-47ce-8920-92cfe99aea61 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:02:42.441802 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b699cf9f-3ddb-47ce-8920-92cfe99aea61 service cinder] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:02:42.452346 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 927/1875] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Aug 30 02:02:42 2022] GET /identity/v3/auth/tokens => generated 3329 bytes in 322 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 02:02:42.453183 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b699cf9f-3ddb-47ce-8920-92cfe99aea61 service cinder] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:02:42.465768 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b699cf9f-3ddb-47ce-8920-92cfe99aea61 service cinder] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:02:42.475249 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-034d725d-8a20-4940-97b9-c91e845d60f4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:02:42.476699 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-034d725d-8a20-4940-97b9-c91e845d60f4 admin admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:02:42.476939 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-034d725d-8a20-4940-97b9-c91e845d60f4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:02:42.477262 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-034d725d-8a20-4940-97b9-c91e845d60f4 admin admin] PATH_INFO: `/v3/roles` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:02:42.478806 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-034d725d-8a20-4940-97b9-c91e845d60f4 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:02:42.481052 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-034d725d-8a20-4940-97b9-c91e845d60f4 admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:02:42.492427 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 928/1876] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Aug 30 02:02:42 2022] GET /identity/v3/roles => generated 1420 bytes in 40 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 02:02:42.496487 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-034d725d-8a20-4940-97b9-c91e845d60f4 admin admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:02:42.506154 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-034d725d-8a20-4940-97b9-c91e845d60f4 admin admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:02:42.512429 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c736c453-924c-4e8b-8f5c-9a3e39d3d935 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:02:42.513385 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c736c453-924c-4e8b-8f5c-9a3e39d3d935 admin admin] REQUEST_METHOD: `PUT` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:02:42.513813 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c736c453-924c-4e8b-8f5c-9a3e39d3d935 admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:02:42.519951 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c736c453-924c-4e8b-8f5c-9a3e39d3d935 admin admin] PATH_INFO: `/v3/projects/83602d77255e4422a6d3bcb34da64871/users/88392549905845498b89f2d5d9f5f36b/roles/ead5fac86a0747ea8823a82efcea8ed7` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:02:42.534792 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c736c453-924c-4e8b-8f5c-9a3e39d3d935 admin admin] RBAC: Authorizing `identity:create_grant(project_id=83602d77255e4422a6d3bcb34da64871, user_id=88392549905845498b89f2d5d9f5f36b, role_id=ead5fac86a0747ea8823a82efcea8ed7)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:02:42.538057 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c736c453-924c-4e8b-8f5c-9a3e39d3d935 admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:02:42.583019 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 949/1877] 10.222.0.27 () {60 vars in 1144 bytes} [Tue Aug 30 02:02:42 2022] POST /identity/v3/users => generated 291 bytes in 183 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 02:02:42.594306 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1eeb4cb4-c6a7-4fb2-95fa-1ad4e04e904b admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:02:42.599187 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1eeb4cb4-c6a7-4fb2-95fa-1ad4e04e904b admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:02:42.603861 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ee303e0-f098-435e-914b-7c2a1c167b8c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:02:42.604470 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ee303e0-f098-435e-914b-7c2a1c167b8c admin admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:02:42.604756 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ee303e0-f098-435e-914b-7c2a1c167b8c admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:02:42.605291 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ee303e0-f098-435e-914b-7c2a1c167b8c admin admin] PATH_INFO: `/v3/roles` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:02:42.605851 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ee303e0-f098-435e-914b-7c2a1c167b8c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:02:42.607358 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ee303e0-f098-435e-914b-7c2a1c167b8c admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:02:42.618023 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 950/1878] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Aug 30 02:02:42 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 30 02:02:42.618919 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ee303e0-f098-435e-914b-7c2a1c167b8c admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:02:42.620634 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-4ee303e0-f098-435e-914b-7c2a1c167b8c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:02:42.637102 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ee303e0-f098-435e-914b-7c2a1c167b8c admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:02:42.642590 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d8db5cc-2526-46ea-9078-33ea7de2b6f1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:02:42.643668 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d8db5cc-2526-46ea-9078-33ea7de2b6f1 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:02:42.643668 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d8db5cc-2526-46ea-9078-33ea7de2b6f1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:02:42.644027 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d8db5cc-2526-46ea-9078-33ea7de2b6f1 admin admin] PATH_INFO: `/v3/projects/cfde2d9f016e46839128fe2158629c25` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:02:42.645198 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d8db5cc-2526-46ea-9078-33ea7de2b6f1 admin admin] RBAC: Authorizing `identity:delete_project(project_id=cfde2d9f016e46839128fe2158629c25)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:02:42.647153 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d8db5cc-2526-46ea-9078-33ea7de2b6f1 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:02:42.717435 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 929/1879] 10.222.0.27 () {60 vars in 1483 bytes} [Tue Aug 30 02:02:42 2022] PUT /identity/v3/projects/83602d77255e4422a6d3bcb34da64871/users/88392549905845498b89f2d5d9f5f36b/roles/ead5fac86a0747ea8823a82efcea8ed7 => generated 0 bytes in 221 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 30 02:02:42.721328 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c736c453-924c-4e8b-8f5c-9a3e39d3d935 admin admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:02:42.775045 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-7d8db5cc-2526-46ea-9078-33ea7de2b6f1 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'cfde2d9f016e46839128fe2158629c25', 'request_id': 'req-7d8db5cc-2526-46ea-9078-33ea7de2b6f1'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:02:42.782710 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-7d8db5cc-2526-46ea-9078-33ea7de2b6f1 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'cfde2d9f016e46839128fe2158629c25', 'request_id': 'req-7d8db5cc-2526-46ea-9078-33ea7de2b6f1'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:02:42.789560 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-7d8db5cc-2526-46ea-9078-33ea7de2b6f1 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'cfde2d9f016e46839128fe2158629c25', 'request_id': 'req-7d8db5cc-2526-46ea-9078-33ea7de2b6f1'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:02:42.803418 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c736c453-924c-4e8b-8f5c-9a3e39d3d935 admin admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:02:42.865877 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f4166fb-b976-4f23-819b-d95d3e816347 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:02:42.866642 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f4166fb-b976-4f23-819b-d95d3e816347 admin admin] REQUEST_METHOD: `PUT` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:02:42.867099 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f4166fb-b976-4f23-819b-d95d3e816347 admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:02:42.867372 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f4166fb-b976-4f23-819b-d95d3e816347 admin admin] PATH_INFO: `/v3/projects/48f825e944db4b0ab1275fa8f914f469/users/1b9607db086d4578b94e53e11d42b34d/roles/ead5fac86a0747ea8823a82efcea8ed7` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:02:42.874802 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:02:42.887412 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f4166fb-b976-4f23-819b-d95d3e816347 admin admin] RBAC: Authorizing `identity:create_grant(project_id=48f825e944db4b0ab1275fa8f914f469, user_id=1b9607db086d4578b94e53e11d42b34d, role_id=ead5fac86a0747ea8823a82efcea8ed7)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:02:42.890063 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f4166fb-b976-4f23-819b-d95d3e816347 admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:02:42.903220 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-7d8db5cc-2526-46ea-9078-33ea7de2b6f1 admin admin] The token cache is being invalidate because project cfde2d9f016e46839128fe2158629c25 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81577) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 30 02:02:42.904387 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-7d8db5cc-2526-46ea-9078-33ea7de2b6f1 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:02:42.905070 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-7d8db5cc-2526-46ea-9078-33ea7de2b6f1 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:02:42.906616 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 951/1880] 10.222.0.27 () {58 vars in 1233 bytes} [Tue Aug 30 02:02:42 2022] DELETE /identity/v3/projects/cfde2d9f016e46839128fe2158629c25 => generated 0 bytes in 288 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 02:02:42.907404 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d8db5cc-2526-46ea-9078-33ea7de2b6f1 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:02:42.908805 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-7d8db5cc-2526-46ea-9078-33ea7de2b6f1 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:02:42.922641 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d8db5cc-2526-46ea-9078-33ea7de2b6f1 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:02:42.927641 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-397a23bc-54c3-4449-a2e6-60dd1124b69b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:02:42.928343 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-397a23bc-54c3-4449-a2e6-60dd1124b69b admin admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:02:42.928692 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-397a23bc-54c3-4449-a2e6-60dd1124b69b admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:02:42.929262 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-397a23bc-54c3-4449-a2e6-60dd1124b69b admin admin] PATH_INFO: `/v3/roles` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:02:42.929911 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-397a23bc-54c3-4449-a2e6-60dd1124b69b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:02:42.931586 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-397a23bc-54c3-4449-a2e6-60dd1124b69b admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:02:42.942797 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 952/1881] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Aug 30 02:02:42 2022] GET /identity/v3/roles => generated 1420 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 02:02:42.947359 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68e18dde-1c45-45fe-926b-e63824739313 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:02:42.947730 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68e18dde-1c45-45fe-926b-e63824739313 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:02:42.947937 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68e18dde-1c45-45fe-926b-e63824739313 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:02:42.961343 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 930/1882] 10.222.0.27 () {60 vars in 1483 bytes} [Tue Aug 30 02:02:42 2022] PUT /identity/v3/projects/48f825e944db4b0ab1275fa8f914f469/users/1b9607db086d4578b94e53e11d42b34d/roles/ead5fac86a0747ea8823a82efcea8ed7 => generated 0 bytes in 239 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 02:02:42.961343 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f4166fb-b976-4f23-819b-d95d3e816347 admin admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:02:42.984958 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-68e18dde-1c45-45fe-926b-e63824739313 None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:02:42.986978 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-68e18dde-1c45-45fe-926b-e63824739313 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:02:43.069658 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 953/1883] 10.222.0.27 () {58 vars in 958 bytes} [Tue Aug 30 02:02:42 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 123 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 02:02:43.076588 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68e18dde-1c45-45fe-926b-e63824739313 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:02:43.078425 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-68e18dde-1c45-45fe-926b-e63824739313 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:02:43.080596 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f4166fb-b976-4f23-819b-d95d3e816347 admin admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:02:43.086808 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f66b9c14-54dc-403a-811c-99e0811f91f2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:02:43.087496 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f66b9c14-54dc-403a-811c-99e0811f91f2 admin admin] REQUEST_METHOD: `PUT` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:02:43.088422 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f66b9c14-54dc-403a-811c-99e0811f91f2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:02:43.088422 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f66b9c14-54dc-403a-811c-99e0811f91f2 admin admin] PATH_INFO: `/v3/projects/83602d77255e4422a6d3bcb34da64871/users/88392549905845498b89f2d5d9f5f36b/roles/ead5fac86a0747ea8823a82efcea8ed7` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:02:43.090353 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f66b9c14-54dc-403a-811c-99e0811f91f2 admin admin] RBAC: Authorizing `identity:create_grant(project_id=83602d77255e4422a6d3bcb34da64871, user_id=88392549905845498b89f2d5d9f5f36b, role_id=ead5fac86a0747ea8823a82efcea8ed7)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:02:43.093430 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68e18dde-1c45-45fe-926b-e63824739313 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:02:43.099968 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3da5f25d-8122-4317-9874-4e1966a4e035 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:02:43.100750 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3da5f25d-8122-4317-9874-4e1966a4e035 admin admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:02:43.101317 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3da5f25d-8122-4317-9874-4e1966a4e035 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:02:43.101444 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3da5f25d-8122-4317-9874-4e1966a4e035 admin admin] PATH_INFO: `/v3/roles` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:02:43.102239 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3da5f25d-8122-4317-9874-4e1966a4e035 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:02:43.105417 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3da5f25d-8122-4317-9874-4e1966a4e035 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:02:43.110249 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f66b9c14-54dc-403a-811c-99e0811f91f2 admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:02:43.116066 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 954/1884] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Aug 30 02:02:43 2022] GET /identity/v3/roles => generated 1420 bytes in 46 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 02:02:43.119381 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04244e81-dee8-4d8f-b095-f383a02ce1bb None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:02:43.119381 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04244e81-dee8-4d8f-b095-f383a02ce1bb None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:02:43.119381 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04244e81-dee8-4d8f-b095-f383a02ce1bb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:02:43.145011 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-04244e81-dee8-4d8f-b095-f383a02ce1bb None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:02:43.146252 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-04244e81-dee8-4d8f-b095-f383a02ce1bb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:02:43.148718 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 931/1885] 10.222.0.27 () {60 vars in 1483 bytes} [Tue Aug 30 02:02:42 2022] PUT /identity/v3/projects/83602d77255e4422a6d3bcb34da64871/users/88392549905845498b89f2d5d9f5f36b/roles/ead5fac86a0747ea8823a82efcea8ed7 => generated 0 bytes in 188 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 02:02:43.152264 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f66b9c14-54dc-403a-811c-99e0811f91f2 admin admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:02:43.268806 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f66b9c14-54dc-403a-811c-99e0811f91f2 admin admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:02:43.281485 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 955/1886] 10.222.0.27 () {58 vars in 958 bytes} [Tue Aug 30 02:02:43 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 165 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 02:02:43.284377 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19d0624a-cac7-4fec-92ba-ae60c15be66f None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:02:43.284652 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7879973e-c1a3-4695-9d04-9d65ca6415d7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:02:43.284877 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19d0624a-cac7-4fec-92ba-ae60c15be66f None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:02:43.285278 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19d0624a-cac7-4fec-92ba-ae60c15be66f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:02:43.285487 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7879973e-c1a3-4695-9d04-9d65ca6415d7 admin admin] REQUEST_METHOD: `PUT` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:02:43.285824 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7879973e-c1a3-4695-9d04-9d65ca6415d7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:02:43.286103 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7879973e-c1a3-4695-9d04-9d65ca6415d7 admin admin] PATH_INFO: `/v3/projects/48f825e944db4b0ab1275fa8f914f469/users/1b9607db086d4578b94e53e11d42b34d/roles/ead5fac86a0747ea8823a82efcea8ed7` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:02:43.290163 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7879973e-c1a3-4695-9d04-9d65ca6415d7 admin admin] RBAC: Authorizing `identity:create_grant(project_id=48f825e944db4b0ab1275fa8f914f469, user_id=1b9607db086d4578b94e53e11d42b34d, role_id=ead5fac86a0747ea8823a82efcea8ed7)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:02:43.308722 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7879973e-c1a3-4695-9d04-9d65ca6415d7 admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:02:43.333693 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 932/1887] 10.222.0.27 () {60 vars in 1483 bytes} [Tue Aug 30 02:02:43 2022] PUT /identity/v3/projects/48f825e944db4b0ab1275fa8f914f469/users/1b9607db086d4578b94e53e11d42b34d/roles/ead5fac86a0747ea8823a82efcea8ed7 => generated 0 bytes in 184 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 02:02:43.335102 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7879973e-c1a3-4695-9d04-9d65ca6415d7 admin admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:02:43.377361 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-19d0624a-cac7-4fec-92ba-ae60c15be66f None None] MFA Rules not processed for user `88392549905845498b89f2d5d9f5f36b`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:02:43.389776 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-19d0624a-cac7-4fec-92ba-ae60c15be66f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:02:43.532334 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7879973e-c1a3-4695-9d04-9d65ca6415d7 admin admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:02:43.544474 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 956/1888] 10.222.0.27 () {58 vars in 958 bytes} [Tue Aug 30 02:02:43 2022] POST /identity/v3/auth/tokens => generated 3307 bytes in 262 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 02:02:43.552558 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50bac450-5e3d-4e44-99ce-26b5639adb26 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:02:43.553270 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50bac450-5e3d-4e44-99ce-26b5639adb26 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:02:43.553883 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50bac450-5e3d-4e44-99ce-26b5639adb26 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:02:43.563697 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20c37821-8e31-4815-a74c-00305141bea3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:02:43.564345 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20c37821-8e31-4815-a74c-00305141bea3 admin admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:02:43.565211 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20c37821-8e31-4815-a74c-00305141bea3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:02:43.565211 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20c37821-8e31-4815-a74c-00305141bea3 admin admin] PATH_INFO: `/v3/domains` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:02:43.568429 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20c37821-8e31-4815-a74c-00305141bea3 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81578) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 02:02:43.568429 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20c37821-8e31-4815-a74c-00305141bea3 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:02:43.570869 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20c37821-8e31-4815-a74c-00305141bea3 admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:02:43.596482 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 933/1889] 10.222.0.27 () {58 vars in 1153 bytes} [Tue Aug 30 02:02:43 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 261 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 02:02:43.600254 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cee5306-83da-46cf-b7d8-615996aeb925 None None] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:02:43.600254 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cee5306-83da-46cf-b7d8-615996aeb925 None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:02:43.600254 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cee5306-83da-46cf-b7d8-615996aeb925 None None] PATH_INFO: `/` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:02:43.600254 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 934/1890] 10.222.0.27 () {54 vars in 921 bytes} [Tue Aug 30 02:02:43 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 02:02:43.600853 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cee5306-83da-46cf-b7d8-615996aeb925 None None] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:02:43.603522 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-8cee5306-83da-46cf-b7d8-615996aeb925 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:02:43.638914 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-50bac450-5e3d-4e44-99ce-26b5639adb26 None None] MFA Rules not processed for user `1b9607db086d4578b94e53e11d42b34d`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:02:43.641703 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-50bac450-5e3d-4e44-99ce-26b5639adb26 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:02:43.725807 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cee5306-83da-46cf-b7d8-615996aeb925 None None] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:02:43.736366 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-281f5407-b407-416b-aece-f311d9e77dfe service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ab936b3767a8417aa80c8c8f1b84b667', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4562f3951fbe465280073e60a11e44e2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:02:43.743150 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-281f5407-b407-416b-aece-f311d9e77dfe service cinder] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:02:43.743150 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-281f5407-b407-416b-aece-f311d9e77dfe service cinder] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:02:43.743150 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-281f5407-b407-416b-aece-f311d9e77dfe service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:02:43.763110 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-281f5407-b407-416b-aece-f311d9e77dfe service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:02:43.764922 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-281f5407-b407-416b-aece-f311d9e77dfe service cinder] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:02:43.769548 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 957/1891] 10.222.0.27 () {58 vars in 958 bytes} [Tue Aug 30 02:02:43 2022] POST /identity/v3/auth/tokens => generated 3313 bytes in 221 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 02:02:43.769548 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50bac450-5e3d-4e44-99ce-26b5639adb26 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:02:43.781011 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 935/1892] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Aug 30 02:02:43 2022] GET /identity/v3/auth/tokens => generated 3307 bytes in 181 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 02:02:43.794966 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a2b591c-2409-433f-bcf4-b1ad0cbf1d9d None None] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:02:43.795164 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a2b591c-2409-433f-bcf4-b1ad0cbf1d9d None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:02:43.795419 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a2b591c-2409-433f-bcf4-b1ad0cbf1d9d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:02:43.796231 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50bac450-5e3d-4e44-99ce-26b5639adb26 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:02:43.811998 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1295dd70-4983-4fdb-809e-10f0cd1cf4cb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:02:43.811998 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1295dd70-4983-4fdb-809e-10f0cd1cf4cb admin admin] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:02:43.812741 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1295dd70-4983-4fdb-809e-10f0cd1cf4cb admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:02:43.813111 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1295dd70-4983-4fdb-809e-10f0cd1cf4cb admin admin] PATH_INFO: `/v3/projects` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:02:43.813885 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1295dd70-4983-4fdb-809e-10f0cd1cf4cb admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:02:43.816024 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1295dd70-4983-4fdb-809e-10f0cd1cf4cb admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:02:43.830775 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.auth.core [None req-5a2b591c-2409-433f-bcf4-b1ad0cbf1d9d None None] MFA Rules not processed for user `d64f2e71361644589bec8a06e6eceed1`. Rule list: `[]` (Enabled: `True`). {{(pid=81578) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:02:43.832160 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-5a2b591c-2409-433f-bcf4-b1ad0cbf1d9d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:02:43.945742 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 936/1893] 10.222.0.27 () {58 vars in 1018 bytes} [Tue Aug 30 02:02:43 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 152 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 02:02:43.946665 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a2b591c-2409-433f-bcf4-b1ad0cbf1d9d None None] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:02:43.948406 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-5a2b591c-2409-433f-bcf4-b1ad0cbf1d9d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:02:43.969705 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a2b591c-2409-433f-bcf4-b1ad0cbf1d9d None None] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:02:43.974066 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a6625ff-c41b-4a54-ab3f-ee03ad52ea72 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ab936b3767a8417aa80c8c8f1b84b667', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4562f3951fbe465280073e60a11e44e2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:02:43.974837 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a6625ff-c41b-4a54-ab3f-ee03ad52ea72 service cinder] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:02:43.975122 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a6625ff-c41b-4a54-ab3f-ee03ad52ea72 service cinder] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:02:43.975393 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a6625ff-c41b-4a54-ab3f-ee03ad52ea72 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:02:43.990442 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a6625ff-c41b-4a54-ab3f-ee03ad52ea72 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:02:43.992710 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a6625ff-c41b-4a54-ab3f-ee03ad52ea72 service cinder] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:02:44.029275 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 958/1894] 10.222.0.27 () {60 vars in 1153 bytes} [Tue Aug 30 02:02:43 2022] POST /identity/v3/projects => generated 373 bytes in 262 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 02:02:44.030403 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1295dd70-4983-4fdb-809e-10f0cd1cf4cb admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:02:44.032757 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 937/1895] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Aug 30 02:02:43 2022] GET /identity/v3/auth/tokens => generated 3313 bytes in 86 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 02:02:44.033206 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-1295dd70-4983-4fdb-809e-10f0cd1cf4cb admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:02:44.033777 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a6625ff-c41b-4a54-ab3f-ee03ad52ea72 service cinder] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:02:44.067264 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a6625ff-c41b-4a54-ab3f-ee03ad52ea72 service cinder] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:02:44.128289 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1295dd70-4983-4fdb-809e-10f0cd1cf4cb admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:02:44.134487 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba097d36-fd02-4bad-8a52-e814e5d32ef4 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd64f2e71361644589bec8a06e6eceed1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4562f3951fbe465280073e60a11e44e2', 'project_domain_id': 'default', 'roles': ['member', 'service', 'admin', 'ResellerAdmin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:02:44.135191 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba097d36-fd02-4bad-8a52-e814e5d32ef4 service nova] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:02:44.135539 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba097d36-fd02-4bad-8a52-e814e5d32ef4 service nova] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:02:44.135884 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba097d36-fd02-4bad-8a52-e814e5d32ef4 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:02:44.146342 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1e60e6b-bd5b-41a3-9383-5e9f4b973557 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:02:44.146558 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba097d36-fd02-4bad-8a52-e814e5d32ef4 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:02:44.146965 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1e60e6b-bd5b-41a3-9383-5e9f4b973557 admin admin] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:02:44.147245 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1e60e6b-bd5b-41a3-9383-5e9f4b973557 admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:02:44.147520 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1e60e6b-bd5b-41a3-9383-5e9f4b973557 admin admin] PATH_INFO: `/v3/users` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:02:44.148155 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1e60e6b-bd5b-41a3-9383-5e9f4b973557 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:02:44.148308 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba097d36-fd02-4bad-8a52-e814e5d32ef4 service nova] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:02:44.150760 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1e60e6b-bd5b-41a3-9383-5e9f4b973557 admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:02:44.160602 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 959/1896] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Aug 30 02:02:44 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 131 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 02:02:44.225409 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba097d36-fd02-4bad-8a52-e814e5d32ef4 service nova] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:02:44.290651 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba097d36-fd02-4bad-8a52-e814e5d32ef4 service nova] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:02:44.296144 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-814adf0a-5727-4ee4-8910-fc79174fb26a service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ab936b3767a8417aa80c8c8f1b84b667', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4562f3951fbe465280073e60a11e44e2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:02:44.296805 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-814adf0a-5727-4ee4-8910-fc79174fb26a service cinder] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:02:44.297163 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-814adf0a-5727-4ee4-8910-fc79174fb26a service cinder] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:02:44.297517 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-814adf0a-5727-4ee4-8910-fc79174fb26a service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:02:44.300159 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-814adf0a-5727-4ee4-8910-fc79174fb26a service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:02:44.323328 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 938/1897] 10.222.0.27 () {60 vars in 1144 bytes} [Tue Aug 30 02:02:44 2022] POST /identity/v3/users => generated 287 bytes in 287 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 02:02:44.327067 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1e60e6b-bd5b-41a3-9383-5e9f4b973557 admin admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:02:44.344875 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1e60e6b-bd5b-41a3-9383-5e9f4b973557 admin admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:02:44.350703 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18d66fcb-973f-47ae-981d-20b1013b34d8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:02:44.351314 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18d66fcb-973f-47ae-981d-20b1013b34d8 admin admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:02:44.351598 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18d66fcb-973f-47ae-981d-20b1013b34d8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:02:44.351867 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18d66fcb-973f-47ae-981d-20b1013b34d8 admin admin] PATH_INFO: `/v3/roles` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:02:44.352686 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18d66fcb-973f-47ae-981d-20b1013b34d8 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:02:44.354194 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18d66fcb-973f-47ae-981d-20b1013b34d8 admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:02:44.374388 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 939/1898] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Aug 30 02:02:44 2022] GET /identity/v3/roles => generated 1420 bytes in 48 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 02:02:44.378503 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18d66fcb-973f-47ae-981d-20b1013b34d8 admin admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:02:44.384362 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-814adf0a-5727-4ee4-8910-fc79174fb26a service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:02:44.386033 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-814adf0a-5727-4ee4-8910-fc79174fb26a service cinder] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:02:44.389978 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18d66fcb-973f-47ae-981d-20b1013b34d8 admin admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:02:44.395348 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f539476a-9882-4a0d-9d58-fb4f539c6be5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:02:44.396159 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f539476a-9882-4a0d-9d58-fb4f539c6be5 admin admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:02:44.396588 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f539476a-9882-4a0d-9d58-fb4f539c6be5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:02:44.397220 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f539476a-9882-4a0d-9d58-fb4f539c6be5 admin admin] PATH_INFO: `/v3/roles` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:02:44.398138 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f539476a-9882-4a0d-9d58-fb4f539c6be5 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:02:44.398632 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 960/1899] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Aug 30 02:02:44 2022] GET /identity/v3/auth/tokens => generated 3240 bytes in 175 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 30 02:02:44.400118 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f539476a-9882-4a0d-9d58-fb4f539c6be5 admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:02:44.418632 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 940/1900] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Aug 30 02:02:44 2022] GET /identity/v3/roles => generated 1420 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 30 02:02:44.420870 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-814adf0a-5727-4ee4-8910-fc79174fb26a service cinder] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:02:44.438593 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-814adf0a-5727-4ee4-8910-fc79174fb26a service cinder] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:02:44.449498 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e5e9e38-5982-42f2-9bbb-f0f4f71ec23d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:02:44.450127 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e5e9e38-5982-42f2-9bbb-f0f4f71ec23d admin admin] REQUEST_METHOD: `PUT` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:02:44.450674 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e5e9e38-5982-42f2-9bbb-f0f4f71ec23d admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:02:44.451167 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e5e9e38-5982-42f2-9bbb-f0f4f71ec23d admin admin] PATH_INFO: `/v3/projects/0c812524985542e49422fc7fd7df1c37/users/5fc9b0dac9e24583a0d76619b6f9e13e/roles/1fb81afcb9f245f3b989add94598a063` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:02:44.474628 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e5e9e38-5982-42f2-9bbb-f0f4f71ec23d admin admin] RBAC: Authorizing `identity:create_grant(project_id=0c812524985542e49422fc7fd7df1c37, user_id=5fc9b0dac9e24583a0d76619b6f9e13e, role_id=1fb81afcb9f245f3b989add94598a063)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:02:44.479695 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e5e9e38-5982-42f2-9bbb-f0f4f71ec23d admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:02:44.567259 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 961/1901] 10.222.0.27 () {60 vars in 1483 bytes} [Tue Aug 30 02:02:44 2022] PUT /identity/v3/projects/0c812524985542e49422fc7fd7df1c37/users/5fc9b0dac9e24583a0d76619b6f9e13e/roles/1fb81afcb9f245f3b989add94598a063 => generated 0 bytes in 149 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 30 02:02:44.583791 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d3eace1-8648-4b92-b20a-7f90c6cf1c98 None None] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:02:44.584570 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d3eace1-8648-4b92-b20a-7f90c6cf1c98 None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:02:44.584570 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d3eace1-8648-4b92-b20a-7f90c6cf1c98 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:02:44.617355 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.auth.core [None req-4d3eace1-8648-4b92-b20a-7f90c6cf1c98 None None] MFA Rules not processed for user `5fc9b0dac9e24583a0d76619b6f9e13e`. Rule list: `[]` (Enabled: `True`). {{(pid=81578) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:02:44.618614 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-4d3eace1-8648-4b92-b20a-7f90c6cf1c98 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:02:44.691018 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 941/1902] 10.222.0.27 () {58 vars in 958 bytes} [Tue Aug 30 02:02:44 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 111 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 02:02:44.695432 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e5e9e38-5982-42f2-9bbb-f0f4f71ec23d admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:02:44.774283 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e5e9e38-5982-42f2-9bbb-f0f4f71ec23d admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:02:44.785860 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9695038e-6374-4548-bc52-9b66346d4f2f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:02:44.786864 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9695038e-6374-4548-bc52-9b66346d4f2f admin admin] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:02:44.786864 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9695038e-6374-4548-bc52-9b66346d4f2f admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:02:44.787169 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9695038e-6374-4548-bc52-9b66346d4f2f admin admin] PATH_INFO: `/v3/projects` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:02:44.788177 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9695038e-6374-4548-bc52-9b66346d4f2f admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:02:44.795309 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9695038e-6374-4548-bc52-9b66346d4f2f admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:02:45.033147 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 962/1903] 10.222.0.27 () {60 vars in 1153 bytes} [Tue Aug 30 02:02:44 2022] POST /identity/v3/projects => generated 373 bytes in 336 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 02:02:45.034593 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d3eace1-8648-4b92-b20a-7f90c6cf1c98 None None] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:02:45.072491 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d3eace1-8648-4b92-b20a-7f90c6cf1c98 None None] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:02:45.180273 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79ca821c-0524-4238-87ce-190be44dc19b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:02:45.184346 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:02:45.185232 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79ca821c-0524-4238-87ce-190be44dc19b admin admin] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:02:45.185636 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79ca821c-0524-4238-87ce-190be44dc19b admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:02:45.185955 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79ca821c-0524-4238-87ce-190be44dc19b admin admin] PATH_INFO: `/v3/users` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:02:45.186822 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79ca821c-0524-4238-87ce-190be44dc19b admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:02:45.189270 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79ca821c-0524-4238-87ce-190be44dc19b admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:02:45.336803 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 942/1904] 10.222.0.27 () {60 vars in 1144 bytes} [Tue Aug 30 02:02:45 2022] POST /identity/v3/users => generated 293 bytes in 303 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 02:02:45.341219 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9695038e-6374-4548-bc52-9b66346d4f2f admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:02:45.355690 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9695038e-6374-4548-bc52-9b66346d4f2f admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:02:45.363100 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75f2f37f-5174-442d-bc6d-40a9d291a79d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:02:45.363100 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75f2f37f-5174-442d-bc6d-40a9d291a79d admin admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:02:45.364114 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75f2f37f-5174-442d-bc6d-40a9d291a79d admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:02:45.364114 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75f2f37f-5174-442d-bc6d-40a9d291a79d admin admin] PATH_INFO: `/v3/roles` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:02:45.364276 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75f2f37f-5174-442d-bc6d-40a9d291a79d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:02:45.366474 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75f2f37f-5174-442d-bc6d-40a9d291a79d admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:02:45.380733 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 963/1905] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Aug 30 02:02:45 2022] GET /identity/v3/roles => generated 1420 bytes in 40 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 30 02:02:45.388418 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79ca821c-0524-4238-87ce-190be44dc19b admin admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:02:45.401572 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79ca821c-0524-4238-87ce-190be44dc19b admin admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:02:45.406636 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e2c1015-6c30-4a32-b9ad-f996a22b456d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:02:45.407543 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e2c1015-6c30-4a32-b9ad-f996a22b456d admin admin] REQUEST_METHOD: `PUT` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:02:45.407932 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e2c1015-6c30-4a32-b9ad-f996a22b456d admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:02:45.408156 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e2c1015-6c30-4a32-b9ad-f996a22b456d admin admin] PATH_INFO: `/v3/projects/14beeece71744334884cd25245c732ff/users/6bb9c9f3aa404faabbabb6427d36fd3e/roles/ead5fac86a0747ea8823a82efcea8ed7` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:02:45.424936 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e2c1015-6c30-4a32-b9ad-f996a22b456d admin admin] RBAC: Authorizing `identity:create_grant(project_id=14beeece71744334884cd25245c732ff, user_id=6bb9c9f3aa404faabbabb6427d36fd3e, role_id=ead5fac86a0747ea8823a82efcea8ed7)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:02:45.427780 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e2c1015-6c30-4a32-b9ad-f996a22b456d admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:02:45.540331 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 943/1906] 10.222.0.27 () {60 vars in 1483 bytes} [Tue Aug 30 02:02:45 2022] PUT /identity/v3/projects/14beeece71744334884cd25245c732ff/users/6bb9c9f3aa404faabbabb6427d36fd3e/roles/ead5fac86a0747ea8823a82efcea8ed7 => generated 0 bytes in 154 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 30 02:02:45.544813 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75f2f37f-5174-442d-bc6d-40a9d291a79d admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:02:45.559329 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75f2f37f-5174-442d-bc6d-40a9d291a79d admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:02:45.652294 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01248e1e-9bc8-45c6-8bf6-81fc81c0a2ba admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:02:45.653439 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01248e1e-9bc8-45c6-8bf6-81fc81c0a2ba admin admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:02:45.654845 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01248e1e-9bc8-45c6-8bf6-81fc81c0a2ba admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:02:45.654845 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01248e1e-9bc8-45c6-8bf6-81fc81c0a2ba admin admin] PATH_INFO: `/v3/roles` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:02:45.655095 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01248e1e-9bc8-45c6-8bf6-81fc81c0a2ba admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:02:45.659551 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01248e1e-9bc8-45c6-8bf6-81fc81c0a2ba admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:02:45.678122 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 964/1907] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Aug 30 02:02:45 2022] GET /identity/v3/roles => generated 1420 bytes in 131 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 02:02:45.679778 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e2c1015-6c30-4a32-b9ad-f996a22b456d admin admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:02:45.691847 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e2c1015-6c30-4a32-b9ad-f996a22b456d admin admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:02:45.699351 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8af4b3c-b824-4a97-8521-6f2802d07583 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:02:45.700872 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8af4b3c-b824-4a97-8521-6f2802d07583 admin admin] REQUEST_METHOD: `PUT` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:02:45.701587 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8af4b3c-b824-4a97-8521-6f2802d07583 admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:02:45.701587 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8af4b3c-b824-4a97-8521-6f2802d07583 admin admin] PATH_INFO: `/v3/projects/14beeece71744334884cd25245c732ff/users/6bb9c9f3aa404faabbabb6427d36fd3e/roles/ead5fac86a0747ea8823a82efcea8ed7` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:02:45.709223 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8af4b3c-b824-4a97-8521-6f2802d07583 admin admin] RBAC: Authorizing `identity:create_grant(project_id=14beeece71744334884cd25245c732ff, user_id=6bb9c9f3aa404faabbabb6427d36fd3e, role_id=ead5fac86a0747ea8823a82efcea8ed7)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:02:45.712121 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8af4b3c-b824-4a97-8521-6f2802d07583 admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:02:45.738060 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 944/1908] 10.222.0.27 () {60 vars in 1483 bytes} [Tue Aug 30 02:02:45 2022] PUT /identity/v3/projects/14beeece71744334884cd25245c732ff/users/6bb9c9f3aa404faabbabb6427d36fd3e/roles/ead5fac86a0747ea8823a82efcea8ed7 => generated 0 bytes in 57 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 02:02:45.754079 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c5ff3d7-f63f-4dcc-a7c2-a065a1480ca5 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:02:45.754079 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c5ff3d7-f63f-4dcc-a7c2-a065a1480ca5 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:02:45.754079 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c5ff3d7-f63f-4dcc-a7c2-a065a1480ca5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:02:45.802775 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-6c5ff3d7-f63f-4dcc-a7c2-a065a1480ca5 None None] MFA Rules not processed for user `6bb9c9f3aa404faabbabb6427d36fd3e`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:02:45.805503 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-6c5ff3d7-f63f-4dcc-a7c2-a065a1480ca5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:02:45.897211 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 965/1909] 10.222.0.27 () {58 vars in 958 bytes} [Tue Aug 30 02:02:45 2022] POST /identity/v3/auth/tokens => generated 3317 bytes in 145 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 02:02:45.905491 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8af4b3c-b824-4a97-8521-6f2802d07583 admin admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:02:45.982528 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8af4b3c-b824-4a97-8521-6f2802d07583 admin admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:02:45.986262 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84d97554-1219-41f7-acda-63e3cfd2f490 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ab936b3767a8417aa80c8c8f1b84b667', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4562f3951fbe465280073e60a11e44e2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:02:45.986868 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84d97554-1219-41f7-acda-63e3cfd2f490 service cinder] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:02:45.987145 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84d97554-1219-41f7-acda-63e3cfd2f490 service cinder] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:02:45.987415 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84d97554-1219-41f7-acda-63e3cfd2f490 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:02:46.052931 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84d97554-1219-41f7-acda-63e3cfd2f490 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:02:46.054539 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84d97554-1219-41f7-acda-63e3cfd2f490 service cinder] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:02:46.067411 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 945/1910] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Aug 30 02:02:45 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 162 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 30 02:02:46.413532 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=89,keystone:INSERT=4,keystone:DELETE=7 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:02:48.040049 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:02:50.927670 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c5ff3d7-f63f-4dcc-a7c2-a065a1480ca5 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:02:51.004222 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c5ff3d7-f63f-4dcc-a7c2-a065a1480ca5 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:02:51.009611 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2818ef2a-44ec-4928-b40f-8c6182380a01 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:02:51.010258 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2818ef2a-44ec-4928-b40f-8c6182380a01 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:02:51.010489 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2818ef2a-44ec-4928-b40f-8c6182380a01 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:02:51.011418 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2818ef2a-44ec-4928-b40f-8c6182380a01 admin admin] PATH_INFO: `/v3/users/5fc9b0dac9e24583a0d76619b6f9e13e` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:02:51.016129 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2818ef2a-44ec-4928-b40f-8c6182380a01 admin admin] RBAC: Authorizing `identity:delete_user(user_id=5fc9b0dac9e24583a0d76619b6f9e13e)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:02:51.018730 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2818ef2a-44ec-4928-b40f-8c6182380a01 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:02:51.092730 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84d97554-1219-41f7-acda-63e3cfd2f490 service cinder] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:02:51.094227 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-84d97554-1219-41f7-acda-63e3cfd2f490 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:02:51.118854 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84d97554-1219-41f7-acda-63e3cfd2f490 service cinder] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:02:51.131052 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b08b5d74-e4df-4e1b-b305-73abcb368fe9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:02:51.131712 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b08b5d74-e4df-4e1b-b305-73abcb368fe9 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:02:51.131987 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b08b5d74-e4df-4e1b-b305-73abcb368fe9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:02:51.132256 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b08b5d74-e4df-4e1b-b305-73abcb368fe9 admin admin] PATH_INFO: `/v3/users/1a7ae71307f642aaa19715f54d72b2ff` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:02:51.133405 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b08b5d74-e4df-4e1b-b305-73abcb368fe9 admin admin] RBAC: Authorizing `identity:delete_user(user_id=1a7ae71307f642aaa19715f54d72b2ff)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:02:51.136937 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b08b5d74-e4df-4e1b-b305-73abcb368fe9 admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:02:51.499476 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-b08b5d74-e4df-4e1b-b305-73abcb368fe9 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '1a7ae71307f642aaa19715f54d72b2ff'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:02:51.500194 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-b08b5d74-e4df-4e1b-b305-73abcb368fe9 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '1a7ae71307f642aaa19715f54d72b2ff'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:02:51.502523 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-2818ef2a-44ec-4928-b40f-8c6182380a01 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '5fc9b0dac9e24583a0d76619b6f9e13e'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:02:51.504731 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-2818ef2a-44ec-4928-b40f-8c6182380a01 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '5fc9b0dac9e24583a0d76619b6f9e13e'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:02:51.505367 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-2818ef2a-44ec-4928-b40f-8c6182380a01 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '5fc9b0dac9e24583a0d76619b6f9e13e'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:02:51.518860 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-2818ef2a-44ec-4928-b40f-8c6182380a01 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '5fc9b0dac9e24583a0d76619b6f9e13e'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:02:51.526279 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-2818ef2a-44ec-4928-b40f-8c6182380a01 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '5fc9b0dac9e24583a0d76619b6f9e13e'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:02:51.606088 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-b08b5d74-e4df-4e1b-b305-73abcb368fe9 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '1a7ae71307f642aaa19715f54d72b2ff'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:02:51.607378 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-b08b5d74-e4df-4e1b-b305-73abcb368fe9 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '1a7ae71307f642aaa19715f54d72b2ff'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:02:51.617640 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-b08b5d74-e4df-4e1b-b305-73abcb368fe9 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '1a7ae71307f642aaa19715f54d72b2ff'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:02:51.639066 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 946/1911] 10.222.0.27 () {58 vars in 1224 bytes} [Tue Aug 30 02:02:51 2022] DELETE /identity/v3/users/1a7ae71307f642aaa19715f54d72b2ff => generated 0 bytes in 547 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 02:02:51.643742 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87056d61-cd9e-4619-93a0-0d673ada5fc4 None None] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:02:51.644244 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87056d61-cd9e-4619-93a0-0d673ada5fc4 None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:02:51.644494 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87056d61-cd9e-4619-93a0-0d673ada5fc4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:02:51.665163 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.auth.core [None req-87056d61-cd9e-4619-93a0-0d673ada5fc4 None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81578) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:02:51.666652 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-87056d61-cd9e-4619-93a0-0d673ada5fc4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:02:51.759820 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 966/1912] 10.222.0.27 () {58 vars in 1224 bytes} [Tue Aug 30 02:02:50 2022] DELETE /identity/v3/users/5fc9b0dac9e24583a0d76619b6f9e13e => generated 0 bytes in 832 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 02:02:51.774793 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b9ce255-6f83-4841-982d-87430e33155e None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:02:51.775187 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b9ce255-6f83-4841-982d-87430e33155e None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:02:51.775559 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b9ce255-6f83-4841-982d-87430e33155e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:02:51.781939 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 947/1913] 10.222.0.27 () {58 vars in 958 bytes} [Tue Aug 30 02:02:51 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 139 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 02:02:51.792767 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87056d61-cd9e-4619-93a0-0d673ada5fc4 None None] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:02:51.794625 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-87056d61-cd9e-4619-93a0-0d673ada5fc4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:02:51.809355 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-9b9ce255-6f83-4841-982d-87430e33155e None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:02:51.809355 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-9b9ce255-6f83-4841-982d-87430e33155e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:02:51.884166 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 967/1914] 10.222.0.27 () {58 vars in 958 bytes} [Tue Aug 30 02:02:51 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 30 02:02:51.884345 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=86,keystone:DELETE=10,keystone:INSERT=4 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:02:51.904605 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b9ce255-6f83-4841-982d-87430e33155e None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:02:51.920629 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87056d61-cd9e-4619-93a0-0d673ada5fc4 None None] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:02:51.967741 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b9ce255-6f83-4841-982d-87430e33155e None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:02:51.972210 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed7bd527-6009-4505-958b-e3db4b2bc8ea service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f3ea1580c56f4814a92bbb2acf542905', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4562f3951fbe465280073e60a11e44e2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:02:51.973164 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed7bd527-6009-4505-958b-e3db4b2bc8ea service neutron] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:02:51.973660 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed7bd527-6009-4505-958b-e3db4b2bc8ea service neutron] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:02:51.974102 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed7bd527-6009-4505-958b-e3db4b2bc8ea service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:02:51.985333 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed7bd527-6009-4505-958b-e3db4b2bc8ea service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:02:51.987617 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed7bd527-6009-4505-958b-e3db4b2bc8ea service neutron] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:02:51.993976 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1945a34-5b0d-4329-acef-3a2ea21f9beb service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f3ea1580c56f4814a92bbb2acf542905', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4562f3951fbe465280073e60a11e44e2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:02:51.995527 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1945a34-5b0d-4329-acef-3a2ea21f9beb service neutron] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:02:51.996364 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1945a34-5b0d-4329-acef-3a2ea21f9beb service neutron] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:02:51.996907 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1945a34-5b0d-4329-acef-3a2ea21f9beb service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:02:52.003439 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 968/1915] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Aug 30 02:02:51 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 100 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 30 02:02:52.008150 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1945a34-5b0d-4329-acef-3a2ea21f9beb service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:02:52.010085 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1945a34-5b0d-4329-acef-3a2ea21f9beb service neutron] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:02:52.022652 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 948/1916] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Aug 30 02:02:51 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 230 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 02:02:52.714558 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed7bd527-6009-4505-958b-e3db4b2bc8ea service neutron] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:02:52.714558 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-ed7bd527-6009-4505-958b-e3db4b2bc8ea service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:02:52.730274 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed7bd527-6009-4505-958b-e3db4b2bc8ea service neutron] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:02:52.735491 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f5bbade-014b-4c9c-9197-7f1fe5372401 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:02:52.736138 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f5bbade-014b-4c9c-9197-7f1fe5372401 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:02:52.744847 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f5bbade-014b-4c9c-9197-7f1fe5372401 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:02:52.744847 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f5bbade-014b-4c9c-9197-7f1fe5372401 admin admin] PATH_INFO: `/v3/projects/0c812524985542e49422fc7fd7df1c37` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:02:52.744847 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f5bbade-014b-4c9c-9197-7f1fe5372401 admin admin] RBAC: Authorizing `identity:delete_project(project_id=0c812524985542e49422fc7fd7df1c37)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:02:52.744847 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f5bbade-014b-4c9c-9197-7f1fe5372401 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:02:52.788219 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:02:52.852665 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1945a34-5b0d-4329-acef-3a2ea21f9beb service neutron] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:02:52.855741 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-b1945a34-5b0d-4329-acef-3a2ea21f9beb service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:02:52.885931 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1945a34-5b0d-4329-acef-3a2ea21f9beb service neutron] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:02:52.903100 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-3f5bbade-014b-4c9c-9197-7f1fe5372401 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '0c812524985542e49422fc7fd7df1c37', 'request_id': 'req-3f5bbade-014b-4c9c-9197-7f1fe5372401'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:02:52.910862 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-3f5bbade-014b-4c9c-9197-7f1fe5372401 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '0c812524985542e49422fc7fd7df1c37', 'request_id': 'req-3f5bbade-014b-4c9c-9197-7f1fe5372401'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:02:52.920134 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-3f5bbade-014b-4c9c-9197-7f1fe5372401 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '0c812524985542e49422fc7fd7df1c37', 'request_id': 'req-3f5bbade-014b-4c9c-9197-7f1fe5372401'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:02:52.947986 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10c6e1ab-2ba2-4da8-a157-8c5678b19fa0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:02:52.948624 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10c6e1ab-2ba2-4da8-a157-8c5678b19fa0 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:02:52.948952 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10c6e1ab-2ba2-4da8-a157-8c5678b19fa0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:02:52.949225 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10c6e1ab-2ba2-4da8-a157-8c5678b19fa0 admin admin] PATH_INFO: `/v3/projects/c18e9b4b4e6542e1a7d3e08f6acea712` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:02:52.950545 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10c6e1ab-2ba2-4da8-a157-8c5678b19fa0 admin admin] RBAC: Authorizing `identity:delete_project(project_id=c18e9b4b4e6542e1a7d3e08f6acea712)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:02:52.953762 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10c6e1ab-2ba2-4da8-a157-8c5678b19fa0 admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:02:52.998759 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-3f5bbade-014b-4c9c-9197-7f1fe5372401 admin admin] The token cache is being invalidate because project 0c812524985542e49422fc7fd7df1c37 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81577) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 30 02:02:52.999315 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-3f5bbade-014b-4c9c-9197-7f1fe5372401 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:02:52.999749 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-3f5bbade-014b-4c9c-9197-7f1fe5372401 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:02:53.001343 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 969/1917] 10.222.0.27 () {58 vars in 1233 bytes} [Tue Aug 30 02:02:52 2022] DELETE /identity/v3/projects/0c812524985542e49422fc7fd7df1c37 => generated 0 bytes in 292 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 02:02:53.002203 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f5bbade-014b-4c9c-9197-7f1fe5372401 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:02:53.003874 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-3f5bbade-014b-4c9c-9197-7f1fe5372401 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:02:53.018370 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f5bbade-014b-4c9c-9197-7f1fe5372401 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:02:53.026869 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbbfc080-f71d-40ab-8ca2-756449dc278c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:02:53.027558 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbbfc080-f71d-40ab-8ca2-756449dc278c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:02:53.027829 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbbfc080-f71d-40ab-8ca2-756449dc278c admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:02:53.028121 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbbfc080-f71d-40ab-8ca2-756449dc278c admin admin] PATH_INFO: `/v3/users/e9101336c65047fea32ed1212dddb547` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:02:53.029556 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbbfc080-f71d-40ab-8ca2-756449dc278c admin admin] RBAC: Authorizing `identity:delete_user(user_id=e9101336c65047fea32ed1212dddb547)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:02:53.031528 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbbfc080-f71d-40ab-8ca2-756449dc278c admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:02:53.075464 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-10c6e1ab-2ba2-4da8-a157-8c5678b19fa0 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'c18e9b4b4e6542e1a7d3e08f6acea712', 'request_id': 'req-10c6e1ab-2ba2-4da8-a157-8c5678b19fa0'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:02:53.081035 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-10c6e1ab-2ba2-4da8-a157-8c5678b19fa0 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'c18e9b4b4e6542e1a7d3e08f6acea712', 'request_id': 'req-10c6e1ab-2ba2-4da8-a157-8c5678b19fa0'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:02:53.087256 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-10c6e1ab-2ba2-4da8-a157-8c5678b19fa0 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'c18e9b4b4e6542e1a7d3e08f6acea712', 'request_id': 'req-10c6e1ab-2ba2-4da8-a157-8c5678b19fa0'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:02:53.211237 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-10c6e1ab-2ba2-4da8-a157-8c5678b19fa0 admin admin] The token cache is being invalidate because project c18e9b4b4e6542e1a7d3e08f6acea712 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81578) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 30 02:02:53.211434 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-10c6e1ab-2ba2-4da8-a157-8c5678b19fa0 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:02:53.212146 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-10c6e1ab-2ba2-4da8-a157-8c5678b19fa0 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:02:53.214454 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 949/1918] 10.222.0.27 () {58 vars in 1233 bytes} [Tue Aug 30 02:02:52 2022] DELETE /identity/v3/projects/c18e9b4b4e6542e1a7d3e08f6acea712 => generated 0 bytes in 362 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 02:02:53.217958 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10c6e1ab-2ba2-4da8-a157-8c5678b19fa0 admin admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:02:53.217958 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-10c6e1ab-2ba2-4da8-a157-8c5678b19fa0 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:02:53.285164 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-fbbfc080-f71d-40ab-8ca2-756449dc278c admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'e9101336c65047fea32ed1212dddb547'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:02:53.285986 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-fbbfc080-f71d-40ab-8ca2-756449dc278c admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'e9101336c65047fea32ed1212dddb547'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:02:53.286689 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-fbbfc080-f71d-40ab-8ca2-756449dc278c admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'e9101336c65047fea32ed1212dddb547'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:02:53.302220 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-fbbfc080-f71d-40ab-8ca2-756449dc278c admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'e9101336c65047fea32ed1212dddb547'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:02:53.310237 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-fbbfc080-f71d-40ab-8ca2-756449dc278c admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'e9101336c65047fea32ed1212dddb547'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:02:53.329009 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10c6e1ab-2ba2-4da8-a157-8c5678b19fa0 admin admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:02:53.333760 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02998b92-b40c-46f5-b1ed-96201c89beb8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:02:53.334426 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02998b92-b40c-46f5-b1ed-96201c89beb8 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:02:53.334708 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02998b92-b40c-46f5-b1ed-96201c89beb8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:02:53.334983 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02998b92-b40c-46f5-b1ed-96201c89beb8 admin admin] PATH_INFO: `/v3/users/6bb9c9f3aa404faabbabb6427d36fd3e` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:02:53.336121 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02998b92-b40c-46f5-b1ed-96201c89beb8 admin admin] RBAC: Authorizing `identity:delete_user(user_id=6bb9c9f3aa404faabbabb6427d36fd3e)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:02:53.338083 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02998b92-b40c-46f5-b1ed-96201c89beb8 admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:02:53.553124 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 970/1919] 10.222.0.27 () {58 vars in 1224 bytes} [Tue Aug 30 02:02:53 2022] DELETE /identity/v3/users/e9101336c65047fea32ed1212dddb547 => generated 0 bytes in 551 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 02:02:53.560673 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbbfc080-f71d-40ab-8ca2-756449dc278c admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:02:53.560673 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-fbbfc080-f71d-40ab-8ca2-756449dc278c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:02:53.622981 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbbfc080-f71d-40ab-8ca2-756449dc278c admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:02:53.627684 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-990d1d6b-246f-44c3-a4b4-8569393544f3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:02:53.628319 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-990d1d6b-246f-44c3-a4b4-8569393544f3 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:02:53.628601 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-990d1d6b-246f-44c3-a4b4-8569393544f3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:02:53.628873 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-990d1d6b-246f-44c3-a4b4-8569393544f3 admin admin] PATH_INFO: `/v3/users/c44667b3e57b4703943113b10b548d73` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:02:53.630028 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-990d1d6b-246f-44c3-a4b4-8569393544f3 admin admin] RBAC: Authorizing `identity:delete_user(user_id=c44667b3e57b4703943113b10b548d73)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:02:53.632004 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-990d1d6b-246f-44c3-a4b4-8569393544f3 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:02:53.700251 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-02998b92-b40c-46f5-b1ed-96201c89beb8 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '6bb9c9f3aa404faabbabb6427d36fd3e'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:02:53.701024 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-02998b92-b40c-46f5-b1ed-96201c89beb8 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '6bb9c9f3aa404faabbabb6427d36fd3e'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:02:53.799369 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-02998b92-b40c-46f5-b1ed-96201c89beb8 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '6bb9c9f3aa404faabbabb6427d36fd3e'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:02:53.800140 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-02998b92-b40c-46f5-b1ed-96201c89beb8 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '6bb9c9f3aa404faabbabb6427d36fd3e'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:02:53.810954 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-02998b92-b40c-46f5-b1ed-96201c89beb8 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '6bb9c9f3aa404faabbabb6427d36fd3e'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:02:53.830212 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 950/1920] 10.222.0.27 () {58 vars in 1224 bytes} [Tue Aug 30 02:02:53 2022] DELETE /identity/v3/users/6bb9c9f3aa404faabbabb6427d36fd3e => generated 0 bytes in 615 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 02:02:53.831657 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-221c90c2-31c8-49f2-b59e-f88b97cc789e None None] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:02:53.832210 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-221c90c2-31c8-49f2-b59e-f88b97cc789e None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:02:53.832306 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-221c90c2-31c8-49f2-b59e-f88b97cc789e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:02:53.852365 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-990d1d6b-246f-44c3-a4b4-8569393544f3 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'c44667b3e57b4703943113b10b548d73'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:02:53.853932 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-990d1d6b-246f-44c3-a4b4-8569393544f3 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'c44667b3e57b4703943113b10b548d73'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:02:53.855571 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-990d1d6b-246f-44c3-a4b4-8569393544f3 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'c44667b3e57b4703943113b10b548d73'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:02:53.857265 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.auth.core [None req-221c90c2-31c8-49f2-b59e-f88b97cc789e None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81578) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:02:53.859085 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-221c90c2-31c8-49f2-b59e-f88b97cc789e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:02:53.873071 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-990d1d6b-246f-44c3-a4b4-8569393544f3 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'c44667b3e57b4703943113b10b548d73'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:02:53.882318 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-990d1d6b-246f-44c3-a4b4-8569393544f3 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'c44667b3e57b4703943113b10b548d73'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:02:53.928144 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 951/1921] 10.222.0.27 () {58 vars in 958 bytes} [Tue Aug 30 02:02:53 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 02:02:53.938400 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-221c90c2-31c8-49f2-b59e-f88b97cc789e None None] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:02:53.939856 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-221c90c2-31c8-49f2-b59e-f88b97cc789e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:02:53.962620 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 971/1922] 10.222.0.27 () {58 vars in 1224 bytes} [Tue Aug 30 02:02:53 2022] DELETE /identity/v3/users/c44667b3e57b4703943113b10b548d73 => generated 0 bytes in 408 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 02:02:54.013540 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-221c90c2-31c8-49f2-b59e-f88b97cc789e None None] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:02:54.079327 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87296b1a-b37b-4867-8c61-dbbc7b22b6f0 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f3ea1580c56f4814a92bbb2acf542905', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4562f3951fbe465280073e60a11e44e2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:02:54.080257 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87296b1a-b37b-4867-8c61-dbbc7b22b6f0 service neutron] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:02:54.081691 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87296b1a-b37b-4867-8c61-dbbc7b22b6f0 service neutron] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:02:54.082156 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87296b1a-b37b-4867-8c61-dbbc7b22b6f0 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:02:54.159284 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87296b1a-b37b-4867-8c61-dbbc7b22b6f0 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:02:54.160731 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87296b1a-b37b-4867-8c61-dbbc7b22b6f0 service neutron] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:02:54.171273 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 952/1923] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Aug 30 02:02:53 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 233 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 30 02:02:54.418973 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-990d1d6b-246f-44c3-a4b4-8569393544f3 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:02:54.420535 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-990d1d6b-246f-44c3-a4b4-8569393544f3 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:02:54.440056 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-990d1d6b-246f-44c3-a4b4-8569393544f3 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:02:54.447225 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b01b7bab-196c-405d-820a-3aebfc3902e7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:02:54.447926 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b01b7bab-196c-405d-820a-3aebfc3902e7 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:02:54.448840 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b01b7bab-196c-405d-820a-3aebfc3902e7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:02:54.449647 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b01b7bab-196c-405d-820a-3aebfc3902e7 admin admin] PATH_INFO: `/v3/projects/14beeece71744334884cd25245c732ff` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:02:54.452354 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b01b7bab-196c-405d-820a-3aebfc3902e7 admin admin] RBAC: Authorizing `identity:delete_project(project_id=14beeece71744334884cd25245c732ff)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:02:54.456113 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b01b7bab-196c-405d-820a-3aebfc3902e7 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:02:54.540382 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87296b1a-b37b-4867-8c61-dbbc7b22b6f0 service neutron] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:02:54.542257 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-87296b1a-b37b-4867-8c61-dbbc7b22b6f0 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:02:54.560520 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87296b1a-b37b-4867-8c61-dbbc7b22b6f0 service neutron] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:02:54.591646 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d3be7c5-8b1d-421c-8be9-2ea250bd6d30 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:02:54.591646 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d3be7c5-8b1d-421c-8be9-2ea250bd6d30 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:02:54.591646 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d3be7c5-8b1d-421c-8be9-2ea250bd6d30 admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:02:54.591646 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d3be7c5-8b1d-421c-8be9-2ea250bd6d30 admin admin] PATH_INFO: `/v3/projects/74b66f9413bc48939877bad0a15cd509` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:02:54.592369 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d3be7c5-8b1d-421c-8be9-2ea250bd6d30 admin admin] RBAC: Authorizing `identity:delete_project(project_id=74b66f9413bc48939877bad0a15cd509)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:02:54.592369 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d3be7c5-8b1d-421c-8be9-2ea250bd6d30 admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:02:54.665886 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-b01b7bab-196c-405d-820a-3aebfc3902e7 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '14beeece71744334884cd25245c732ff', 'request_id': 'req-b01b7bab-196c-405d-820a-3aebfc3902e7'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:02:54.677434 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-b01b7bab-196c-405d-820a-3aebfc3902e7 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '14beeece71744334884cd25245c732ff', 'request_id': 'req-b01b7bab-196c-405d-820a-3aebfc3902e7'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:02:54.688933 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-b01b7bab-196c-405d-820a-3aebfc3902e7 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '14beeece71744334884cd25245c732ff', 'request_id': 'req-b01b7bab-196c-405d-820a-3aebfc3902e7'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:02:54.759068 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-3d3be7c5-8b1d-421c-8be9-2ea250bd6d30 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '74b66f9413bc48939877bad0a15cd509', 'request_id': 'req-3d3be7c5-8b1d-421c-8be9-2ea250bd6d30'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:02:54.769409 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-3d3be7c5-8b1d-421c-8be9-2ea250bd6d30 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '74b66f9413bc48939877bad0a15cd509', 'request_id': 'req-3d3be7c5-8b1d-421c-8be9-2ea250bd6d30'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:02:54.774490 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-3d3be7c5-8b1d-421c-8be9-2ea250bd6d30 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '74b66f9413bc48939877bad0a15cd509', 'request_id': 'req-3d3be7c5-8b1d-421c-8be9-2ea250bd6d30'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:02:54.821979 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-b01b7bab-196c-405d-820a-3aebfc3902e7 admin admin] The token cache is being invalidate because project 14beeece71744334884cd25245c732ff was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81577) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 30 02:02:54.822330 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-b01b7bab-196c-405d-820a-3aebfc3902e7 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:02:54.823609 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-b01b7bab-196c-405d-820a-3aebfc3902e7 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:02:54.833416 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 972/1924] 10.222.0.27 () {58 vars in 1233 bytes} [Tue Aug 30 02:02:54 2022] DELETE /identity/v3/projects/14beeece71744334884cd25245c732ff => generated 0 bytes in 412 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 02:02:54.844327 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a715cc10-3bb0-4503-991a-b41bbe63f965 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:02:54.844953 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a715cc10-3bb0-4503-991a-b41bbe63f965 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:02:54.845330 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a715cc10-3bb0-4503-991a-b41bbe63f965 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:02:54.880248 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-a715cc10-3bb0-4503-991a-b41bbe63f965 None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:02:54.881710 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-a715cc10-3bb0-4503-991a-b41bbe63f965 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:02:54.955746 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-3d3be7c5-8b1d-421c-8be9-2ea250bd6d30 admin admin] The token cache is being invalidate because project 74b66f9413bc48939877bad0a15cd509 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81578) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 30 02:02:54.960349 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-3d3be7c5-8b1d-421c-8be9-2ea250bd6d30 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:02:54.967389 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-3d3be7c5-8b1d-421c-8be9-2ea250bd6d30 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:02:54.971963 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 953/1925] 10.222.0.27 () {58 vars in 1233 bytes} [Tue Aug 30 02:02:54 2022] DELETE /identity/v3/projects/74b66f9413bc48939877bad0a15cd509 => generated 0 bytes in 431 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 30 02:02:54.975239 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d3be7c5-8b1d-421c-8be9-2ea250bd6d30 admin admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:02:54.977524 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-3d3be7c5-8b1d-421c-8be9-2ea250bd6d30 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:02:54.984117 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 973/1926] 10.222.0.27 () {58 vars in 958 bytes} [Tue Aug 30 02:02:54 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 146 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 02:02:54.995570 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a056d0b-37d0-43d4-a76f-b36c33fc13f1 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:02:54.997175 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a056d0b-37d0-43d4-a76f-b36c33fc13f1 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:02:54.998524 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a056d0b-37d0-43d4-a76f-b36c33fc13f1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:02:55.015417 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d3be7c5-8b1d-421c-8be9-2ea250bd6d30 admin admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:02:55.021850 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aeb314db-b233-46a2-950e-59ea80c60d49 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:02:55.022918 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aeb314db-b233-46a2-950e-59ea80c60d49 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:02:55.023570 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aeb314db-b233-46a2-950e-59ea80c60d49 admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:02:55.024257 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aeb314db-b233-46a2-950e-59ea80c60d49 admin admin] PATH_INFO: `/v3/users/d91a28c01d4f4ce1bc8b011e0d142d34` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:02:55.025775 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aeb314db-b233-46a2-950e-59ea80c60d49 admin admin] RBAC: Authorizing `identity:delete_user(user_id=d91a28c01d4f4ce1bc8b011e0d142d34)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:02:55.029252 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-9a056d0b-37d0-43d4-a76f-b36c33fc13f1 None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:02:55.030109 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aeb314db-b233-46a2-950e-59ea80c60d49 admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:02:55.030552 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-9a056d0b-37d0-43d4-a76f-b36c33fc13f1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:02:55.043476 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 974/1927] 10.222.0.27 () {58 vars in 958 bytes} [Tue Aug 30 02:02:54 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 50 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 30 02:02:55.048964 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a056d0b-37d0-43d4-a76f-b36c33fc13f1 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:02:55.051515 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-9a056d0b-37d0-43d4-a76f-b36c33fc13f1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:02:55.068691 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a056d0b-37d0-43d4-a76f-b36c33fc13f1 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:02:55.079373 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5626ce31-1aee-42b3-b2e9-8c070a49c444 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:02:55.079373 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5626ce31-1aee-42b3-b2e9-8c070a49c444 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:02:55.079373 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5626ce31-1aee-42b3-b2e9-8c070a49c444 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:02:55.079373 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5626ce31-1aee-42b3-b2e9-8c070a49c444 admin admin] PATH_INFO: `/v3/projects/bb6da553928244dfb44d34212725fce6` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:02:55.079766 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5626ce31-1aee-42b3-b2e9-8c070a49c444 admin admin] RBAC: Authorizing `identity:delete_project(project_id=bb6da553928244dfb44d34212725fce6)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:02:55.079766 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5626ce31-1aee-42b3-b2e9-8c070a49c444 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:02:55.410207 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-5626ce31-1aee-42b3-b2e9-8c070a49c444 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'bb6da553928244dfb44d34212725fce6', 'request_id': 'req-5626ce31-1aee-42b3-b2e9-8c070a49c444'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:02:55.410524 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-aeb314db-b233-46a2-950e-59ea80c60d49 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'd91a28c01d4f4ce1bc8b011e0d142d34'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:02:55.411203 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-aeb314db-b233-46a2-950e-59ea80c60d49 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'd91a28c01d4f4ce1bc8b011e0d142d34'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:02:55.419769 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-5626ce31-1aee-42b3-b2e9-8c070a49c444 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'bb6da553928244dfb44d34212725fce6', 'request_id': 'req-5626ce31-1aee-42b3-b2e9-8c070a49c444'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:02:55.429256 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-5626ce31-1aee-42b3-b2e9-8c070a49c444 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'bb6da553928244dfb44d34212725fce6', 'request_id': 'req-5626ce31-1aee-42b3-b2e9-8c070a49c444'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:02:55.490794 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-aeb314db-b233-46a2-950e-59ea80c60d49 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'd91a28c01d4f4ce1bc8b011e0d142d34'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:02:55.491481 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-aeb314db-b233-46a2-950e-59ea80c60d49 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'd91a28c01d4f4ce1bc8b011e0d142d34'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:02:55.499784 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-aeb314db-b233-46a2-950e-59ea80c60d49 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'd91a28c01d4f4ce1bc8b011e0d142d34'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:02:55.518769 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 954/1928] 10.222.0.27 () {58 vars in 1224 bytes} [Tue Aug 30 02:02:54 2022] DELETE /identity/v3/users/d91a28c01d4f4ce1bc8b011e0d142d34 => generated 0 bytes in 544 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 02:02:55.519663 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aeb314db-b233-46a2-950e-59ea80c60d49 admin admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:02:55.520964 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-aeb314db-b233-46a2-950e-59ea80c60d49 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:02:55.597079 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aeb314db-b233-46a2-950e-59ea80c60d49 admin admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:02:55.607286 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-5626ce31-1aee-42b3-b2e9-8c070a49c444 admin admin] The token cache is being invalidate because project bb6da553928244dfb44d34212725fce6 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81577) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 30 02:02:55.612557 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5982aad-4d9e-480c-b637-5d74efe2a82f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:02:55.612712 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-5626ce31-1aee-42b3-b2e9-8c070a49c444 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:02:55.612925 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5982aad-4d9e-480c-b637-5d74efe2a82f admin admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:02:55.612925 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5982aad-4d9e-480c-b637-5d74efe2a82f admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:02:55.613088 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-5626ce31-1aee-42b3-b2e9-8c070a49c444 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:02:55.613163 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5982aad-4d9e-480c-b637-5d74efe2a82f admin admin] PATH_INFO: `/v3/domains` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:02:55.613163 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5982aad-4d9e-480c-b637-5d74efe2a82f admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81578) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 02:02:55.613163 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5982aad-4d9e-480c-b637-5d74efe2a82f admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:02:55.613335 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 975/1929] 10.222.0.27 () {58 vars in 1233 bytes} [Tue Aug 30 02:02:55 2022] DELETE /identity/v3/projects/bb6da553928244dfb44d34212725fce6 => generated 0 bytes in 567 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 02:02:55.613410 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5982aad-4d9e-480c-b637-5d74efe2a82f admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:02:55.615370 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5626ce31-1aee-42b3-b2e9-8c070a49c444 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:02:55.618018 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-5626ce31-1aee-42b3-b2e9-8c070a49c444 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:02:55.628936 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 955/1930] 10.222.0.27 () {58 vars in 1153 bytes} [Tue Aug 30 02:02:55 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 107 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 02:02:55.630868 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5982aad-4d9e-480c-b637-5d74efe2a82f admin admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:02:55.632202 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-b5982aad-4d9e-480c-b637-5d74efe2a82f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:02:55.649200 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5982aad-4d9e-480c-b637-5d74efe2a82f admin admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:02:55.662501 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c60cbf9-b7ab-4c2c-b58f-5dd291501129 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:02:55.663233 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c60cbf9-b7ab-4c2c-b58f-5dd291501129 admin admin] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:02:55.668021 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c60cbf9-b7ab-4c2c-b58f-5dd291501129 admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:02:55.668021 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c60cbf9-b7ab-4c2c-b58f-5dd291501129 admin admin] PATH_INFO: `/v3/projects` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:02:55.668021 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c60cbf9-b7ab-4c2c-b58f-5dd291501129 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:02:55.668021 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c60cbf9-b7ab-4c2c-b58f-5dd291501129 admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:02:55.728056 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5626ce31-1aee-42b3-b2e9-8c070a49c444 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:02:55.735146 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e6e08ad-804b-4826-a749-d83d54bc0400 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:02:55.736329 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e6e08ad-804b-4826-a749-d83d54bc0400 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:02:55.736669 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e6e08ad-804b-4826-a749-d83d54bc0400 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:02:55.736950 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e6e08ad-804b-4826-a749-d83d54bc0400 admin admin] PATH_INFO: `/v3/users/1b9607db086d4578b94e53e11d42b34d` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:02:55.738310 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e6e08ad-804b-4826-a749-d83d54bc0400 admin admin] RBAC: Authorizing `identity:delete_user(user_id=1b9607db086d4578b94e53e11d42b34d)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:02:55.742097 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e6e08ad-804b-4826-a749-d83d54bc0400 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:02:55.826728 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 956/1931] 10.222.0.27 () {60 vars in 1153 bytes} [Tue Aug 30 02:02:55 2022] POST /identity/v3/projects => generated 367 bytes in 197 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 02:02:55.831312 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c60cbf9-b7ab-4c2c-b58f-5dd291501129 admin admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:02:55.930448 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c60cbf9-b7ab-4c2c-b58f-5dd291501129 admin admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:02:55.937069 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47159dbe-d5e3-4512-9180-ffdc42fd2294 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:02:55.937759 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47159dbe-d5e3-4512-9180-ffdc42fd2294 admin admin] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:02:55.938373 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47159dbe-d5e3-4512-9180-ffdc42fd2294 admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:02:55.938499 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47159dbe-d5e3-4512-9180-ffdc42fd2294 admin admin] PATH_INFO: `/v3/users` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:02:55.939397 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47159dbe-d5e3-4512-9180-ffdc42fd2294 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:02:55.941479 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47159dbe-d5e3-4512-9180-ffdc42fd2294 admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:02:56.079777 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-7e6e08ad-804b-4826-a749-d83d54bc0400 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '1b9607db086d4578b94e53e11d42b34d'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:02:56.080482 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-7e6e08ad-804b-4826-a749-d83d54bc0400 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '1b9607db086d4578b94e53e11d42b34d'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:02:56.081103 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-7e6e08ad-804b-4826-a749-d83d54bc0400 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '1b9607db086d4578b94e53e11d42b34d'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:02:56.099338 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-7e6e08ad-804b-4826-a749-d83d54bc0400 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '1b9607db086d4578b94e53e11d42b34d'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:02:56.113528 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-7e6e08ad-804b-4826-a749-d83d54bc0400 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '1b9607db086d4578b94e53e11d42b34d'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:02:56.154355 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 957/1932] 10.222.0.27 () {60 vars in 1144 bytes} [Tue Aug 30 02:02:55 2022] POST /identity/v3/users => generated 284 bytes in 323 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 02:02:56.161602 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47159dbe-d5e3-4512-9180-ffdc42fd2294 admin admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:02:56.163199 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-47159dbe-d5e3-4512-9180-ffdc42fd2294 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:02:56.175540 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47159dbe-d5e3-4512-9180-ffdc42fd2294 admin admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:02:56.180587 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4eca41b-9024-416f-9dfe-9dcb5c26762d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:02:56.181392 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4eca41b-9024-416f-9dfe-9dcb5c26762d admin admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:02:56.181702 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4eca41b-9024-416f-9dfe-9dcb5c26762d admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:02:56.181970 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4eca41b-9024-416f-9dfe-9dcb5c26762d admin admin] PATH_INFO: `/v3/roles` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:02:56.183113 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4eca41b-9024-416f-9dfe-9dcb5c26762d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:02:56.184842 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4eca41b-9024-416f-9dfe-9dcb5c26762d admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:02:56.200089 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 958/1933] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Aug 30 02:02:56 2022] GET /identity/v3/roles => generated 1420 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 30 02:02:56.214180 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4eca41b-9024-416f-9dfe-9dcb5c26762d admin admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:02:56.225151 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4eca41b-9024-416f-9dfe-9dcb5c26762d admin admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:02:56.230073 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-616306d9-3a25-4bd2-b0b8-97b9c7fa2d04 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:02:56.230681 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-616306d9-3a25-4bd2-b0b8-97b9c7fa2d04 admin admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:02:56.231075 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-616306d9-3a25-4bd2-b0b8-97b9c7fa2d04 admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:02:56.231349 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-616306d9-3a25-4bd2-b0b8-97b9c7fa2d04 admin admin] PATH_INFO: `/v3/roles` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:02:56.232325 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-616306d9-3a25-4bd2-b0b8-97b9c7fa2d04 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:02:56.233962 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-616306d9-3a25-4bd2-b0b8-97b9c7fa2d04 admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:02:56.244547 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 959/1934] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Aug 30 02:02:56 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 02:02:56.249396 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-616306d9-3a25-4bd2-b0b8-97b9c7fa2d04 admin admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:02:56.264250 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-616306d9-3a25-4bd2-b0b8-97b9c7fa2d04 admin admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:02:56.277484 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46dc272d-e759-4be4-9462-1eb1767bc476 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:02:56.278177 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46dc272d-e759-4be4-9462-1eb1767bc476 admin admin] REQUEST_METHOD: `PUT` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:02:56.278464 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46dc272d-e759-4be4-9462-1eb1767bc476 admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:02:56.278744 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46dc272d-e759-4be4-9462-1eb1767bc476 admin admin] PATH_INFO: `/v3/projects/df11f3896394474499599fda2fe2289b/users/358c3080807d412485ad516b644f5b47/roles/1fb81afcb9f245f3b989add94598a063` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:02:56.296968 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46dc272d-e759-4be4-9462-1eb1767bc476 admin admin] RBAC: Authorizing `identity:create_grant(project_id=df11f3896394474499599fda2fe2289b, user_id=358c3080807d412485ad516b644f5b47, role_id=1fb81afcb9f245f3b989add94598a063)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:02:56.299675 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46dc272d-e759-4be4-9462-1eb1767bc476 admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:02:56.303930 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 976/1935] 10.222.0.27 () {58 vars in 1224 bytes} [Tue Aug 30 02:02:55 2022] DELETE /identity/v3/users/1b9607db086d4578b94e53e11d42b34d => generated 0 bytes in 689 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 02:02:56.307791 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e6e08ad-804b-4826-a749-d83d54bc0400 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:02:56.310146 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-7e6e08ad-804b-4826-a749-d83d54bc0400 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:02:56.391311 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e6e08ad-804b-4826-a749-d83d54bc0400 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:02:56.395904 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3a2ad0a-eebe-44b4-b2b3-1f6722c0284c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:02:56.396548 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3a2ad0a-eebe-44b4-b2b3-1f6722c0284c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:02:56.396828 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3a2ad0a-eebe-44b4-b2b3-1f6722c0284c admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:02:56.397096 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3a2ad0a-eebe-44b4-b2b3-1f6722c0284c admin admin] PATH_INFO: `/v3/projects/dfd11c6fe40545d3b647cafeedf448e8` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:02:56.398816 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3a2ad0a-eebe-44b4-b2b3-1f6722c0284c admin admin] RBAC: Authorizing `identity:delete_project(project_id=dfd11c6fe40545d3b647cafeedf448e8)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:02:56.401982 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3a2ad0a-eebe-44b4-b2b3-1f6722c0284c admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:02:56.431962 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 960/1936] 10.222.0.27 () {60 vars in 1483 bytes} [Tue Aug 30 02:02:56 2022] PUT /identity/v3/projects/df11f3896394474499599fda2fe2289b/users/358c3080807d412485ad516b644f5b47/roles/1fb81afcb9f245f3b989add94598a063 => generated 0 bytes in 174 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 02:02:56.434983 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7793fce9-0507-4061-8f75-897b0706b048 None None] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:02:56.435323 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7793fce9-0507-4061-8f75-897b0706b048 None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:02:56.435606 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7793fce9-0507-4061-8f75-897b0706b048 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:02:56.472809 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.auth.core [None req-7793fce9-0507-4061-8f75-897b0706b048 None None] MFA Rules not processed for user `358c3080807d412485ad516b644f5b47`. Rule list: `[]` (Enabled: `True`). {{(pid=81578) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:02:56.474125 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-7793fce9-0507-4061-8f75-897b0706b048 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:02:56.554087 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-b3a2ad0a-eebe-44b4-b2b3-1f6722c0284c admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'dfd11c6fe40545d3b647cafeedf448e8', 'request_id': 'req-b3a2ad0a-eebe-44b4-b2b3-1f6722c0284c'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:02:56.554087 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-b3a2ad0a-eebe-44b4-b2b3-1f6722c0284c admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'dfd11c6fe40545d3b647cafeedf448e8', 'request_id': 'req-b3a2ad0a-eebe-44b4-b2b3-1f6722c0284c'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:02:56.562387 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-b3a2ad0a-eebe-44b4-b2b3-1f6722c0284c admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'dfd11c6fe40545d3b647cafeedf448e8', 'request_id': 'req-b3a2ad0a-eebe-44b4-b2b3-1f6722c0284c'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:02:56.579653 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 961/1937] 10.222.0.27 () {58 vars in 958 bytes} [Tue Aug 30 02:02:56 2022] POST /identity/v3/auth/tokens => generated 3244 bytes in 146 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 02:02:56.582746 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=96,keystone:INSERT=1,keystone:DELETE=3 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:02:56.597802 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7793fce9-0507-4061-8f75-897b0706b048 None None] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:02:56.599567 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-7793fce9-0507-4061-8f75-897b0706b048 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:02:56.712439 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-b3a2ad0a-eebe-44b4-b2b3-1f6722c0284c admin admin] The token cache is being invalidate because project dfd11c6fe40545d3b647cafeedf448e8 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81577) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 30 02:02:56.712738 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-b3a2ad0a-eebe-44b4-b2b3-1f6722c0284c admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:02:56.713790 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-b3a2ad0a-eebe-44b4-b2b3-1f6722c0284c admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:02:56.715692 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 977/1938] 10.222.0.27 () {58 vars in 1233 bytes} [Tue Aug 30 02:02:56 2022] DELETE /identity/v3/projects/dfd11c6fe40545d3b647cafeedf448e8 => generated 0 bytes in 411 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 02:02:56.726507 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36081c6e-4f25-4ea6-9365-699e924092b6 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:02:56.726802 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36081c6e-4f25-4ea6-9365-699e924092b6 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:02:56.727072 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36081c6e-4f25-4ea6-9365-699e924092b6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:02:56.732399 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7793fce9-0507-4061-8f75-897b0706b048 None None] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:02:56.770216 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-36081c6e-4f25-4ea6-9365-699e924092b6 None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:02:56.785651 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-36081c6e-4f25-4ea6-9365-699e924092b6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:02:56.817931 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9ed881d-828a-4aa6-9ed3-9c2286ddad21 service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3caff62093d748f48596574836e12e42', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4562f3951fbe465280073e60a11e44e2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:02:56.819005 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9ed881d-828a-4aa6-9ed3-9c2286ddad21 service glance] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:02:56.819322 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9ed881d-828a-4aa6-9ed3-9c2286ddad21 service glance] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:02:56.819600 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9ed881d-828a-4aa6-9ed3-9c2286ddad21 service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:02:56.821526 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-c9ed881d-828a-4aa6-9ed3-9c2286ddad21 service glance] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:02:56.873649 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 978/1939] 10.222.0.27 () {58 vars in 958 bytes} [Tue Aug 30 02:02:56 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 136 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 02:02:56.875206 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d13ec735-31cb-41ab-a395-3269dcd654e7 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:02:56.875494 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d13ec735-31cb-41ab-a395-3269dcd654e7 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:02:56.875771 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d13ec735-31cb-41ab-a395-3269dcd654e7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:02:56.896038 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-d13ec735-31cb-41ab-a395-3269dcd654e7 None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:02:56.896806 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9ed881d-828a-4aa6-9ed3-9c2286ddad21 service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:02:56.898279 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9ed881d-828a-4aa6-9ed3-9c2286ddad21 service glance] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:02:56.898564 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-d13ec735-31cb-41ab-a395-3269dcd654e7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:02:56.908981 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 962/1940] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Aug 30 02:02:56 2022] GET /identity/v3/auth/tokens => generated 3244 bytes in 312 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 30 02:02:56.915658 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 979/1941] 10.222.0.27 () {58 vars in 958 bytes} [Tue Aug 30 02:02:56 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 41 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 30 02:02:56.921164 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9ed881d-828a-4aa6-9ed3-9c2286ddad21 service glance] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:02:56.936796 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9ed881d-828a-4aa6-9ed3-9c2286ddad21 service glance] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:02:56.954866 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-016c5c25-696f-4f3f-8c01-0b540a03ceee admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:02:56.954866 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-016c5c25-696f-4f3f-8c01-0b540a03ceee admin admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:02:56.954866 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-016c5c25-696f-4f3f-8c01-0b540a03ceee admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:02:56.954866 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-016c5c25-696f-4f3f-8c01-0b540a03ceee admin admin] PATH_INFO: `/v3/domains` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:02:56.955145 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-016c5c25-696f-4f3f-8c01-0b540a03ceee admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81578) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 02:02:56.955145 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-016c5c25-696f-4f3f-8c01-0b540a03ceee admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:02:56.955145 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-016c5c25-696f-4f3f-8c01-0b540a03ceee admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:02:56.961576 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 963/1942] 10.222.0.27 () {58 vars in 1153 bytes} [Tue Aug 30 02:02:56 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 02:02:56.962601 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d13ec735-31cb-41ab-a395-3269dcd654e7 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:02:56.972685 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d13ec735-31cb-41ab-a395-3269dcd654e7 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:02:56.977837 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3261bf1d-dab9-4e73-94fa-1f1974cc6a35 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:02:56.978333 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3261bf1d-dab9-4e73-94fa-1f1974cc6a35 admin admin] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:02:56.978612 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3261bf1d-dab9-4e73-94fa-1f1974cc6a35 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:02:56.978885 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3261bf1d-dab9-4e73-94fa-1f1974cc6a35 admin admin] PATH_INFO: `/v3/projects` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:02:56.980056 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3261bf1d-dab9-4e73-94fa-1f1974cc6a35 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:02:56.982111 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3261bf1d-dab9-4e73-94fa-1f1974cc6a35 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:02:57.054089 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-016c5c25-696f-4f3f-8c01-0b540a03ceee admin admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:02:57.059488 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-016c5c25-696f-4f3f-8c01-0b540a03ceee admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:02:57.078844 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-016c5c25-696f-4f3f-8c01-0b540a03ceee admin admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:02:57.086710 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-390fea69-0ecd-4610-a023-6fae972d9969 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:02:57.088127 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-390fea69-0ecd-4610-a023-6fae972d9969 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:02:57.088449 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-390fea69-0ecd-4610-a023-6fae972d9969 admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:02:57.088801 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-390fea69-0ecd-4610-a023-6fae972d9969 admin admin] PATH_INFO: `/v3/projects/48f825e944db4b0ab1275fa8f914f469` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:02:57.090948 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-390fea69-0ecd-4610-a023-6fae972d9969 admin admin] RBAC: Authorizing `identity:delete_project(project_id=48f825e944db4b0ab1275fa8f914f469)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:02:57.093770 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-390fea69-0ecd-4610-a023-6fae972d9969 admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:02:57.177701 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 980/1943] 10.222.0.27 () {60 vars in 1153 bytes} [Tue Aug 30 02:02:56 2022] POST /identity/v3/projects => generated 383 bytes in 216 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 30 02:02:57.186858 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3261bf1d-dab9-4e73-94fa-1f1974cc6a35 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:02:57.255359 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-390fea69-0ecd-4610-a023-6fae972d9969 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '48f825e944db4b0ab1275fa8f914f469', 'request_id': 'req-390fea69-0ecd-4610-a023-6fae972d9969'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:02:57.268225 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-390fea69-0ecd-4610-a023-6fae972d9969 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '48f825e944db4b0ab1275fa8f914f469', 'request_id': 'req-390fea69-0ecd-4610-a023-6fae972d9969'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:02:57.283555 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-390fea69-0ecd-4610-a023-6fae972d9969 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '48f825e944db4b0ab1275fa8f914f469', 'request_id': 'req-390fea69-0ecd-4610-a023-6fae972d9969'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:02:57.318141 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3261bf1d-dab9-4e73-94fa-1f1974cc6a35 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:02:57.366670 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:02:57.401889 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-168149f8-470d-47ec-a7f7-d4cb074288a8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:02:57.402595 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-168149f8-470d-47ec-a7f7-d4cb074288a8 admin admin] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:02:57.402975 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-168149f8-470d-47ec-a7f7-d4cb074288a8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:02:57.403823 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-168149f8-470d-47ec-a7f7-d4cb074288a8 admin admin] PATH_INFO: `/v3/users` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:02:57.404569 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-168149f8-470d-47ec-a7f7-d4cb074288a8 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:02:57.407143 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-168149f8-470d-47ec-a7f7-d4cb074288a8 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:02:57.418606 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-390fea69-0ecd-4610-a023-6fae972d9969 admin admin] The token cache is being invalidate because project 48f825e944db4b0ab1275fa8f914f469 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81578) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 30 02:02:57.419080 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-390fea69-0ecd-4610-a023-6fae972d9969 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:02:57.423769 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-390fea69-0ecd-4610-a023-6fae972d9969 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:02:57.423769 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 964/1944] 10.222.0.27 () {58 vars in 1233 bytes} [Tue Aug 30 02:02:57 2022] DELETE /identity/v3/projects/48f825e944db4b0ab1275fa8f914f469 => generated 0 bytes in 368 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 30 02:02:57.427546 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0be1c084-8cb1-4935-8320-440650e7ad58 None None] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:02:57.427833 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0be1c084-8cb1-4935-8320-440650e7ad58 None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:02:57.428106 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0be1c084-8cb1-4935-8320-440650e7ad58 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:02:57.453296 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.auth.core [None req-0be1c084-8cb1-4935-8320-440650e7ad58 None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81578) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:02:57.454548 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-0be1c084-8cb1-4935-8320-440650e7ad58 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:02:57.479028 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 965/1945] 10.222.0.27 () {58 vars in 958 bytes} [Tue Aug 30 02:02:57 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 52 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 02:02:57.486823 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e48ca7c1-a2f8-4732-adba-87e2b67350d7 None None] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:02:57.487106 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e48ca7c1-a2f8-4732-adba-87e2b67350d7 None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:02:57.487383 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e48ca7c1-a2f8-4732-adba-87e2b67350d7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:02:57.516443 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.auth.core [None req-e48ca7c1-a2f8-4732-adba-87e2b67350d7 None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81578) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:02:57.517904 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-e48ca7c1-a2f8-4732-adba-87e2b67350d7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:02:57.526357 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 966/1946] 10.222.0.27 () {58 vars in 958 bytes} [Tue Aug 30 02:02:57 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 41 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 30 02:02:57.530706 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e48ca7c1-a2f8-4732-adba-87e2b67350d7 None None] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:02:57.540097 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e48ca7c1-a2f8-4732-adba-87e2b67350d7 None None] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:02:57.544741 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52e2b437-8188-487b-ba80-56e2f82e729c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:02:57.545516 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52e2b437-8188-487b-ba80-56e2f82e729c admin admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:02:57.545869 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52e2b437-8188-487b-ba80-56e2f82e729c admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:02:57.546205 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52e2b437-8188-487b-ba80-56e2f82e729c admin admin] PATH_INFO: `/v3/domains` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:02:57.546969 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52e2b437-8188-487b-ba80-56e2f82e729c admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81578) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 02:02:57.547330 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52e2b437-8188-487b-ba80-56e2f82e729c admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:02:57.549067 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52e2b437-8188-487b-ba80-56e2f82e729c admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:02:57.561805 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 967/1947] 10.222.0.27 () {58 vars in 1153 bytes} [Tue Aug 30 02:02:57 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 30 02:02:57.565959 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52e2b437-8188-487b-ba80-56e2f82e729c admin admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:02:57.576976 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52e2b437-8188-487b-ba80-56e2f82e729c admin admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:02:57.581591 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf39936b-ac96-47a1-ab12-1aab86d33f22 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:02:57.582186 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf39936b-ac96-47a1-ab12-1aab86d33f22 admin admin] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:02:57.582465 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf39936b-ac96-47a1-ab12-1aab86d33f22 admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:02:57.582736 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf39936b-ac96-47a1-ab12-1aab86d33f22 admin admin] PATH_INFO: `/v3/projects` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:02:57.583369 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf39936b-ac96-47a1-ab12-1aab86d33f22 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:02:57.585241 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf39936b-ac96-47a1-ab12-1aab86d33f22 admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:02:57.595081 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 981/1948] 10.222.0.27 () {60 vars in 1144 bytes} [Tue Aug 30 02:02:57 2022] POST /identity/v3/users => generated 292 bytes in 414 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 30 02:02:57.604708 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-168149f8-470d-47ec-a7f7-d4cb074288a8 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:02:57.606554 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-168149f8-470d-47ec-a7f7-d4cb074288a8 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:02:57.630136 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-168149f8-470d-47ec-a7f7-d4cb074288a8 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:02:57.634764 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dff435bf-4f0f-4fc3-9352-60095e810ab0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:02:57.636144 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dff435bf-4f0f-4fc3-9352-60095e810ab0 admin admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:02:57.636617 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dff435bf-4f0f-4fc3-9352-60095e810ab0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:02:57.637005 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dff435bf-4f0f-4fc3-9352-60095e810ab0 admin admin] PATH_INFO: `/v3/roles` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:02:57.637696 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dff435bf-4f0f-4fc3-9352-60095e810ab0 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:02:57.639284 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dff435bf-4f0f-4fc3-9352-60095e810ab0 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:02:57.652351 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 982/1949] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Aug 30 02:02:57 2022] GET /identity/v3/roles => generated 1420 bytes in 49 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 02:02:57.656531 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dff435bf-4f0f-4fc3-9352-60095e810ab0 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:02:57.668748 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dff435bf-4f0f-4fc3-9352-60095e810ab0 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:02:57.676451 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d462df9-a29d-4afa-9c03-52dc41c73e49 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:02:57.677182 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d462df9-a29d-4afa-9c03-52dc41c73e49 admin admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:02:57.677563 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d462df9-a29d-4afa-9c03-52dc41c73e49 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:02:57.677905 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d462df9-a29d-4afa-9c03-52dc41c73e49 admin admin] PATH_INFO: `/v3/roles` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:02:57.678615 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d462df9-a29d-4afa-9c03-52dc41c73e49 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:02:57.680291 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d462df9-a29d-4afa-9c03-52dc41c73e49 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:02:57.694247 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 983/1950] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Aug 30 02:02:57 2022] GET /identity/v3/roles => generated 1420 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 02:02:57.698444 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d462df9-a29d-4afa-9c03-52dc41c73e49 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:02:57.706495 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 968/1951] 10.222.0.27 () {60 vars in 1153 bytes} [Tue Aug 30 02:02:57 2022] POST /identity/v3/projects => generated 393 bytes in 141 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 30 02:02:57.712652 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf39936b-ac96-47a1-ab12-1aab86d33f22 admin admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:02:57.721739 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d462df9-a29d-4afa-9c03-52dc41c73e49 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:02:57.783553 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf39936b-ac96-47a1-ab12-1aab86d33f22 admin admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:02:57.788393 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-648aeff3-fe6e-47be-a240-a55f24251e4b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:02:57.789009 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-648aeff3-fe6e-47be-a240-a55f24251e4b admin admin] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:02:57.789289 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-648aeff3-fe6e-47be-a240-a55f24251e4b admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:02:57.790146 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-648aeff3-fe6e-47be-a240-a55f24251e4b admin admin] PATH_INFO: `/v3/users` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:02:57.791106 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-648aeff3-fe6e-47be-a240-a55f24251e4b admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:02:57.793908 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-648aeff3-fe6e-47be-a240-a55f24251e4b admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:02:57.800452 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e79ada9-ff63-4e87-a4a1-2370bcadecb1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:02:57.801934 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e79ada9-ff63-4e87-a4a1-2370bcadecb1 admin admin] REQUEST_METHOD: `PUT` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:02:57.801934 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e79ada9-ff63-4e87-a4a1-2370bcadecb1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:02:57.802148 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e79ada9-ff63-4e87-a4a1-2370bcadecb1 admin admin] PATH_INFO: `/v3/projects/676b687ac2524ed7b343db1c39a3f985/users/0f0497f48ec9464babfdbec23b6e236f/roles/1fb81afcb9f245f3b989add94598a063` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:02:57.819402 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e79ada9-ff63-4e87-a4a1-2370bcadecb1 admin admin] RBAC: Authorizing `identity:create_grant(project_id=676b687ac2524ed7b343db1c39a3f985, user_id=0f0497f48ec9464babfdbec23b6e236f, role_id=1fb81afcb9f245f3b989add94598a063)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:02:57.822150 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e79ada9-ff63-4e87-a4a1-2370bcadecb1 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:02:57.913657 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 984/1952] 10.222.0.27 () {60 vars in 1483 bytes} [Tue Aug 30 02:02:57 2022] PUT /identity/v3/projects/676b687ac2524ed7b343db1c39a3f985/users/0f0497f48ec9464babfdbec23b6e236f/roles/1fb81afcb9f245f3b989add94598a063 => generated 0 bytes in 216 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 30 02:02:57.917753 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 969/1953] 10.222.0.27 () {60 vars in 1144 bytes} [Tue Aug 30 02:02:57 2022] POST /identity/v3/users => generated 297 bytes in 202 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 02:02:57.919367 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e79ada9-ff63-4e87-a4a1-2370bcadecb1 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:02:57.924548 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44b1c361-ef89-41f7-8bee-c95be75d489a None None] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:02:57.924834 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44b1c361-ef89-41f7-8bee-c95be75d489a None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:02:57.925108 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44b1c361-ef89-41f7-8bee-c95be75d489a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:02:57.955050 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.auth.core [None req-44b1c361-ef89-41f7-8bee-c95be75d489a None None] MFA Rules not processed for user `0f0497f48ec9464babfdbec23b6e236f`. Rule list: `[]` (Enabled: `True`). {{(pid=81578) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:02:57.956332 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-44b1c361-ef89-41f7-8bee-c95be75d489a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:02:58.036097 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e79ada9-ff63-4e87-a4a1-2370bcadecb1 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:02:58.041697 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2e28ba4-f5e1-467e-a02b-2cc7b54eff12 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:02:58.042447 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2e28ba4-f5e1-467e-a02b-2cc7b54eff12 admin admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:02:58.042732 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2e28ba4-f5e1-467e-a02b-2cc7b54eff12 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:02:58.043007 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2e28ba4-f5e1-467e-a02b-2cc7b54eff12 admin admin] PATH_INFO: `/v3/roles` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:02:58.043750 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2e28ba4-f5e1-467e-a02b-2cc7b54eff12 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:02:58.045367 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2e28ba4-f5e1-467e-a02b-2cc7b54eff12 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:02:58.049042 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 970/1954] 10.222.0.27 () {58 vars in 958 bytes} [Tue Aug 30 02:02:57 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 126 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 02:02:58.062698 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 985/1955] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Aug 30 02:02:57 2022] GET /identity/v3/roles => generated 1420 bytes in 141 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 02:02:58.066331 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2e28ba4-f5e1-467e-a02b-2cc7b54eff12 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:02:58.066450 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44b1c361-ef89-41f7-8bee-c95be75d489a None None] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:02:58.069323 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-f2e28ba4-f5e1-467e-a02b-2cc7b54eff12 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:02:58.078802 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44b1c361-ef89-41f7-8bee-c95be75d489a None None] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:02:58.084033 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f45842d6-c666-430d-be7c-a3de4691ba38 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:02:58.084648 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f45842d6-c666-430d-be7c-a3de4691ba38 admin admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:02:58.084928 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f45842d6-c666-430d-be7c-a3de4691ba38 admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:02:58.085215 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f45842d6-c666-430d-be7c-a3de4691ba38 admin admin] PATH_INFO: `/v3/roles` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:02:58.085940 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f45842d6-c666-430d-be7c-a3de4691ba38 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:02:58.087528 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f45842d6-c666-430d-be7c-a3de4691ba38 admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:02:58.102346 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 971/1956] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Aug 30 02:02:58 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 02:02:58.104966 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f45842d6-c666-430d-be7c-a3de4691ba38 admin admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:02:58.116808 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f45842d6-c666-430d-be7c-a3de4691ba38 admin admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:02:58.127932 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a60803e-7be9-4970-a465-d12f91a7d63b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:02:58.128628 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a60803e-7be9-4970-a465-d12f91a7d63b admin admin] REQUEST_METHOD: `PUT` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:02:58.129028 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a60803e-7be9-4970-a465-d12f91a7d63b admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:02:58.133806 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a60803e-7be9-4970-a465-d12f91a7d63b admin admin] PATH_INFO: `/v3/projects/476f9e3d07ff4b688527b640a0ad7256/users/5bc0855d88664e548a50bc6305dfdb49/roles/1fb81afcb9f245f3b989add94598a063` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:02:58.144356 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2e28ba4-f5e1-467e-a02b-2cc7b54eff12 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:02:58.148505 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-249c87e1-28b6-4252-9268-69ce51790307 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ab936b3767a8417aa80c8c8f1b84b667', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4562f3951fbe465280073e60a11e44e2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:02:58.149249 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-249c87e1-28b6-4252-9268-69ce51790307 service cinder] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:02:58.149858 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-249c87e1-28b6-4252-9268-69ce51790307 service cinder] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:02:58.150191 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-249c87e1-28b6-4252-9268-69ce51790307 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:02:58.161340 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-249c87e1-28b6-4252-9268-69ce51790307 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:02:58.163044 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-249c87e1-28b6-4252-9268-69ce51790307 service cinder] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:02:58.173025 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a60803e-7be9-4970-a465-d12f91a7d63b admin admin] RBAC: Authorizing `identity:create_grant(project_id=476f9e3d07ff4b688527b640a0ad7256, user_id=5bc0855d88664e548a50bc6305dfdb49, role_id=1fb81afcb9f245f3b989add94598a063)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:02:58.173952 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 986/1957] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Aug 30 02:02:58 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 108 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 02:02:58.176330 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a60803e-7be9-4970-a465-d12f91a7d63b admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:02:58.315226 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 972/1958] 10.222.0.27 () {60 vars in 1483 bytes} [Tue Aug 30 02:02:58 2022] PUT /identity/v3/projects/476f9e3d07ff4b688527b640a0ad7256/users/5bc0855d88664e548a50bc6305dfdb49/roles/1fb81afcb9f245f3b989add94598a063 => generated 0 bytes in 202 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 02:02:58.317949 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee8ba825-af7c-44ec-b0cd-30c17bf482af None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:02:58.318450 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee8ba825-af7c-44ec-b0cd-30c17bf482af None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:02:58.318909 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee8ba825-af7c-44ec-b0cd-30c17bf482af None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:02:58.380633 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-ee8ba825-af7c-44ec-b0cd-30c17bf482af None None] MFA Rules not processed for user `5bc0855d88664e548a50bc6305dfdb49`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:02:58.381897 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-ee8ba825-af7c-44ec-b0cd-30c17bf482af None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:02:58.491989 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 987/1959] 10.222.0.27 () {58 vars in 958 bytes} [Tue Aug 30 02:02:58 2022] POST /identity/v3/auth/tokens => generated 3270 bytes in 175 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 02:02:58.498565 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a60803e-7be9-4970-a465-d12f91a7d63b admin admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:02:58.635026 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a60803e-7be9-4970-a465-d12f91a7d63b admin admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:02:58.640321 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fe6c77d-836f-452c-bb90-a7f21c93b96d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:02:58.641045 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fe6c77d-836f-452c-bb90-a7f21c93b96d admin admin] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:02:58.641626 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fe6c77d-836f-452c-bb90-a7f21c93b96d admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:02:58.642034 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fe6c77d-836f-452c-bb90-a7f21c93b96d admin admin] PATH_INFO: `/v3/projects` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:02:58.642663 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fe6c77d-836f-452c-bb90-a7f21c93b96d admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:02:58.644659 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fe6c77d-836f-452c-bb90-a7f21c93b96d admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:02:58.793883 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 973/1960] 10.222.0.27 () {60 vars in 1153 bytes} [Tue Aug 30 02:02:58 2022] POST /identity/v3/projects => generated 391 bytes in 296 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 02:02:58.798781 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee8ba825-af7c-44ec-b0cd-30c17bf482af None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:02:58.814894 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee8ba825-af7c-44ec-b0cd-30c17bf482af None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:02:58.914949 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-767e3fa1-f865-4a88-befa-e157c3ef5436 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:02:58.918895 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-767e3fa1-f865-4a88-befa-e157c3ef5436 admin admin] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:02:58.918895 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-767e3fa1-f865-4a88-befa-e157c3ef5436 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:02:58.918895 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-767e3fa1-f865-4a88-befa-e157c3ef5436 admin admin] PATH_INFO: `/v3/users` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:02:58.918895 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-767e3fa1-f865-4a88-befa-e157c3ef5436 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:02:58.922466 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-767e3fa1-f865-4a88-befa-e157c3ef5436 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:02:59.116990 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fe6c77d-836f-452c-bb90-a7f21c93b96d admin admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:02:59.124964 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 988/1961] 10.222.0.27 () {60 vars in 1144 bytes} [Tue Aug 30 02:02:58 2022] POST /identity/v3/users => generated 302 bytes in 326 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 02:02:59.129905 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fe6c77d-836f-452c-bb90-a7f21c93b96d admin admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:02:59.135330 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1735a69c-8a27-460f-b51c-3bc08f006048 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:02:59.136388 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1735a69c-8a27-460f-b51c-3bc08f006048 admin admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:02:59.136844 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1735a69c-8a27-460f-b51c-3bc08f006048 admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:02:59.137279 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1735a69c-8a27-460f-b51c-3bc08f006048 admin admin] PATH_INFO: `/v3/roles` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:02:59.140830 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1735a69c-8a27-460f-b51c-3bc08f006048 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:02:59.140830 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1735a69c-8a27-460f-b51c-3bc08f006048 admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:02:59.151734 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 974/1962] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Aug 30 02:02:59 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 02:02:59.156063 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-767e3fa1-f865-4a88-befa-e157c3ef5436 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:02:59.167259 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-767e3fa1-f865-4a88-befa-e157c3ef5436 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:02:59.172615 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cf7babd-04e4-43a4-8aec-073ec972db32 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:02:59.173295 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cf7babd-04e4-43a4-8aec-073ec972db32 admin admin] REQUEST_METHOD: `PUT` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:02:59.173742 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cf7babd-04e4-43a4-8aec-073ec972db32 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:02:59.173937 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cf7babd-04e4-43a4-8aec-073ec972db32 admin admin] PATH_INFO: `/v3/projects/a34963efe84f4775a418fdc92c100606/users/1a43058cf7e54449828a43731fa71d6a/roles/ead5fac86a0747ea8823a82efcea8ed7` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:02:59.199138 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2cf7babd-04e4-43a4-8aec-073ec972db32 admin admin] RBAC: Authorizing `identity:create_grant(project_id=a34963efe84f4775a418fdc92c100606, user_id=1a43058cf7e54449828a43731fa71d6a, role_id=ead5fac86a0747ea8823a82efcea8ed7)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:02:59.202134 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2cf7babd-04e4-43a4-8aec-073ec972db32 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:02:59.316101 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 989/1963] 10.222.0.27 () {60 vars in 1483 bytes} [Tue Aug 30 02:02:59 2022] PUT /identity/v3/projects/a34963efe84f4775a418fdc92c100606/users/1a43058cf7e54449828a43731fa71d6a/roles/ead5fac86a0747ea8823a82efcea8ed7 => generated 0 bytes in 160 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 30 02:02:59.320089 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1735a69c-8a27-460f-b51c-3bc08f006048 admin admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:02:59.330693 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1735a69c-8a27-460f-b51c-3bc08f006048 admin admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:02:59.390152 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed0a8b3d-37ef-492e-958f-e61c81bb20ac admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:02:59.390791 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed0a8b3d-37ef-492e-958f-e61c81bb20ac admin admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:02:59.391074 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed0a8b3d-37ef-492e-958f-e61c81bb20ac admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:02:59.391346 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed0a8b3d-37ef-492e-958f-e61c81bb20ac admin admin] PATH_INFO: `/v3/roles` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:02:59.391996 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed0a8b3d-37ef-492e-958f-e61c81bb20ac admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:02:59.393628 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed0a8b3d-37ef-492e-958f-e61c81bb20ac admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:02:59.404147 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 975/1964] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Aug 30 02:02:59 2022] GET /identity/v3/roles => generated 1420 bytes in 85 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 02:02:59.408443 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cf7babd-04e4-43a4-8aec-073ec972db32 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:02:59.421940 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cf7babd-04e4-43a4-8aec-073ec972db32 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:02:59.427348 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2986b5e-be25-40c8-b6d7-361326e14778 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:02:59.428413 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2986b5e-be25-40c8-b6d7-361326e14778 admin admin] REQUEST_METHOD: `PUT` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:02:59.428770 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2986b5e-be25-40c8-b6d7-361326e14778 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:02:59.430326 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:02:59.430808 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2986b5e-be25-40c8-b6d7-361326e14778 admin admin] PATH_INFO: `/v3/projects/a34963efe84f4775a418fdc92c100606/users/1a43058cf7e54449828a43731fa71d6a/roles/ead5fac86a0747ea8823a82efcea8ed7` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:02:59.434054 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2986b5e-be25-40c8-b6d7-361326e14778 admin admin] RBAC: Authorizing `identity:create_grant(project_id=a34963efe84f4775a418fdc92c100606, user_id=1a43058cf7e54449828a43731fa71d6a, role_id=ead5fac86a0747ea8823a82efcea8ed7)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:02:59.436869 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2986b5e-be25-40c8-b6d7-361326e14778 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:02:59.448558 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 990/1965] 10.222.0.27 () {60 vars in 1483 bytes} [Tue Aug 30 02:02:59 2022] PUT /identity/v3/projects/a34963efe84f4775a418fdc92c100606/users/1a43058cf7e54449828a43731fa71d6a/roles/ead5fac86a0747ea8823a82efcea8ed7 => generated 0 bytes in 41 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 02:02:59.460130 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06580b61-6481-49c7-8026-5df52f93b035 None None] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:02:59.460472 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06580b61-6481-49c7-8026-5df52f93b035 None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:02:59.460818 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06580b61-6481-49c7-8026-5df52f93b035 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:02:59.494122 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.auth.core [None req-06580b61-6481-49c7-8026-5df52f93b035 None None] MFA Rules not processed for user `1a43058cf7e54449828a43731fa71d6a`. Rule list: `[]` (Enabled: `True`). {{(pid=81578) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:02:59.495417 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-06580b61-6481-49c7-8026-5df52f93b035 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:02:59.567045 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 976/1966] 10.222.0.27 () {58 vars in 958 bytes} [Tue Aug 30 02:02:59 2022] POST /identity/v3/auth/tokens => generated 3335 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 02:02:59.578136 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2986b5e-be25-40c8-b6d7-361326e14778 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:02:59.640642 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2986b5e-be25-40c8-b6d7-361326e14778 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:02:59.644723 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45c8ca10-f373-44c1-b671-b48d00d7b9c4 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ab936b3767a8417aa80c8c8f1b84b667', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4562f3951fbe465280073e60a11e44e2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:02:59.645258 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45c8ca10-f373-44c1-b671-b48d00d7b9c4 service cinder] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:02:59.645567 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45c8ca10-f373-44c1-b671-b48d00d7b9c4 service cinder] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:02:59.645890 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45c8ca10-f373-44c1-b671-b48d00d7b9c4 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:02:59.656610 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45c8ca10-f373-44c1-b671-b48d00d7b9c4 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:02:59.658084 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45c8ca10-f373-44c1-b671-b48d00d7b9c4 service cinder] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:02:59.668795 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 991/1967] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Aug 30 02:02:59 2022] GET /identity/v3/auth/tokens => generated 3335 bytes in 91 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 02:02:59.726369 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06580b61-6481-49c7-8026-5df52f93b035 None None] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:02:59.727841 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-06580b61-6481-49c7-8026-5df52f93b035 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:02:59.746512 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06580b61-6481-49c7-8026-5df52f93b035 None None] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:02:59.750490 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2207df41-e77d-48ee-99e2-e7079b45b203 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ab936b3767a8417aa80c8c8f1b84b667', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4562f3951fbe465280073e60a11e44e2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:02:59.751159 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2207df41-e77d-48ee-99e2-e7079b45b203 service cinder] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:02:59.751439 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2207df41-e77d-48ee-99e2-e7079b45b203 service cinder] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:02:59.751752 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2207df41-e77d-48ee-99e2-e7079b45b203 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:02:59.805517 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=91,keystone:DELETE=4,keystone:INSERT=5 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:02:59.832893 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2207df41-e77d-48ee-99e2-e7079b45b203 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:02:59.834463 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2207df41-e77d-48ee-99e2-e7079b45b203 service cinder] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:02:59.844567 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 977/1968] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Aug 30 02:02:59 2022] GET /identity/v3/auth/tokens => generated 3270 bytes in 119 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 02:03:00.553779 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45c8ca10-f373-44c1-b671-b48d00d7b9c4 service cinder] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:03:00.555197 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-45c8ca10-f373-44c1-b671-b48d00d7b9c4 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:03:00.623039 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45c8ca10-f373-44c1-b671-b48d00d7b9c4 service cinder] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:03:00.627582 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f40cb205-3566-4b3d-9426-c4944fb8850a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:03:00.628210 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f40cb205-3566-4b3d-9426-c4944fb8850a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:03:00.628488 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f40cb205-3566-4b3d-9426-c4944fb8850a admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:03:00.628758 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f40cb205-3566-4b3d-9426-c4944fb8850a admin admin] PATH_INFO: `/v3/users/4e8bff45c40f4848b3cd4f32c5ad0871` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:03:00.629923 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f40cb205-3566-4b3d-9426-c4944fb8850a admin admin] RBAC: Authorizing `identity:delete_user(user_id=4e8bff45c40f4848b3cd4f32c5ad0871)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:03:00.631833 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f40cb205-3566-4b3d-9426-c4944fb8850a admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:03:00.848320 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-f40cb205-3566-4b3d-9426-c4944fb8850a admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '4e8bff45c40f4848b3cd4f32c5ad0871'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:03:00.849041 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-f40cb205-3566-4b3d-9426-c4944fb8850a admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '4e8bff45c40f4848b3cd4f32c5ad0871'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:03:00.849678 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-f40cb205-3566-4b3d-9426-c4944fb8850a admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '4e8bff45c40f4848b3cd4f32c5ad0871'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:03:00.863098 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-f40cb205-3566-4b3d-9426-c4944fb8850a admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '4e8bff45c40f4848b3cd4f32c5ad0871'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:03:00.870720 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-f40cb205-3566-4b3d-9426-c4944fb8850a admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '4e8bff45c40f4848b3cd4f32c5ad0871'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:03:00.954187 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 992/1969] 10.222.0.27 () {58 vars in 1224 bytes} [Tue Aug 30 02:03:00 2022] DELETE /identity/v3/users/4e8bff45c40f4848b3cd4f32c5ad0871 => generated 0 bytes in 401 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 02:03:00.958538 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfa77a3c-3abb-4f96-b0a6-138c2fada027 None None] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:03:00.959062 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfa77a3c-3abb-4f96-b0a6-138c2fada027 None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:03:00.959170 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfa77a3c-3abb-4f96-b0a6-138c2fada027 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:03:00.979715 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.auth.core [None req-cfa77a3c-3abb-4f96-b0a6-138c2fada027 None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81578) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:03:00.981303 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-cfa77a3c-3abb-4f96-b0a6-138c2fada027 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:03:01.013639 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:03:01.044528 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 978/1970] 10.222.0.27 () {58 vars in 958 bytes} [Tue Aug 30 02:03:00 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 02:03:01.053310 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f40cb205-3566-4b3d-9426-c4944fb8850a admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:03:01.054749 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-f40cb205-3566-4b3d-9426-c4944fb8850a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:03:01.114407 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f40cb205-3566-4b3d-9426-c4944fb8850a admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:03:01.118461 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66d52c95-d2e7-411c-96ca-ad7163b87e84 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f3ea1580c56f4814a92bbb2acf542905', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4562f3951fbe465280073e60a11e44e2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:03:01.119092 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66d52c95-d2e7-411c-96ca-ad7163b87e84 service neutron] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:03:01.119379 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66d52c95-d2e7-411c-96ca-ad7163b87e84 service neutron] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:03:01.119657 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66d52c95-d2e7-411c-96ca-ad7163b87e84 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:03:01.130033 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66d52c95-d2e7-411c-96ca-ad7163b87e84 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:03:01.131466 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66d52c95-d2e7-411c-96ca-ad7163b87e84 service neutron] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:03:01.141126 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 993/1971] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Aug 30 02:03:01 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 89 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 02:03:01.710658 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfa77a3c-3abb-4f96-b0a6-138c2fada027 None None] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:03:01.711223 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-cfa77a3c-3abb-4f96-b0a6-138c2fada027 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:03:01.723392 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfa77a3c-3abb-4f96-b0a6-138c2fada027 None None] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:03:01.728258 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-449a8d30-6864-4627-be49-fe67a43132d1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:03:01.728993 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-449a8d30-6864-4627-be49-fe67a43132d1 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:03:01.729279 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-449a8d30-6864-4627-be49-fe67a43132d1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:03:01.729588 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-449a8d30-6864-4627-be49-fe67a43132d1 admin admin] PATH_INFO: `/v3/projects/dcef17b6bb014685adcc154bb43c8ec8` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:03:01.730726 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-449a8d30-6864-4627-be49-fe67a43132d1 admin admin] RBAC: Authorizing `identity:delete_project(project_id=dcef17b6bb014685adcc154bb43c8ec8)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:03:01.732623 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-449a8d30-6864-4627-be49-fe67a43132d1 admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:03:01.742711 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66d52c95-d2e7-411c-96ca-ad7163b87e84 service neutron] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:03:01.744890 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-66d52c95-d2e7-411c-96ca-ad7163b87e84 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:03:01.758434 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66d52c95-d2e7-411c-96ca-ad7163b87e84 service neutron] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:03:01.764235 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04233f9f-844d-462d-941a-879038a2ac28 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:03:01.764873 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04233f9f-844d-462d-941a-879038a2ac28 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:03:01.765157 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04233f9f-844d-462d-941a-879038a2ac28 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:03:01.765434 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04233f9f-844d-462d-941a-879038a2ac28 admin admin] PATH_INFO: `/v3/users/0f0497f48ec9464babfdbec23b6e236f` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:03:01.766595 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04233f9f-844d-462d-941a-879038a2ac28 admin admin] RBAC: Authorizing `identity:delete_user(user_id=0f0497f48ec9464babfdbec23b6e236f)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:03:01.768513 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04233f9f-844d-462d-941a-879038a2ac28 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:03:01.867426 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-449a8d30-6864-4627-be49-fe67a43132d1 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'dcef17b6bb014685adcc154bb43c8ec8', 'request_id': 'req-449a8d30-6864-4627-be49-fe67a43132d1'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:03:01.873240 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-449a8d30-6864-4627-be49-fe67a43132d1 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'dcef17b6bb014685adcc154bb43c8ec8', 'request_id': 'req-449a8d30-6864-4627-be49-fe67a43132d1'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:03:01.879674 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-449a8d30-6864-4627-be49-fe67a43132d1 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'dcef17b6bb014685adcc154bb43c8ec8', 'request_id': 'req-449a8d30-6864-4627-be49-fe67a43132d1'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:03:01.993602 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-449a8d30-6864-4627-be49-fe67a43132d1 admin admin] The token cache is being invalidate because project dcef17b6bb014685adcc154bb43c8ec8 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81578) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 30 02:03:01.994160 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-449a8d30-6864-4627-be49-fe67a43132d1 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:03:01.995191 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-449a8d30-6864-4627-be49-fe67a43132d1 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:03:01.996744 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 979/1972] 10.222.0.27 () {58 vars in 1233 bytes} [Tue Aug 30 02:03:01 2022] DELETE /identity/v3/projects/dcef17b6bb014685adcc154bb43c8ec8 => generated 0 bytes in 288 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 02:03:02.001937 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-449a8d30-6864-4627-be49-fe67a43132d1 admin admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:03:02.003521 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-449a8d30-6864-4627-be49-fe67a43132d1 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:03:02.015118 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-04233f9f-844d-462d-941a-879038a2ac28 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '0f0497f48ec9464babfdbec23b6e236f'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:03:02.016098 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-04233f9f-844d-462d-941a-879038a2ac28 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '0f0497f48ec9464babfdbec23b6e236f'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:03:02.017010 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-04233f9f-844d-462d-941a-879038a2ac28 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '0f0497f48ec9464babfdbec23b6e236f'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:03:02.038628 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-04233f9f-844d-462d-941a-879038a2ac28 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '0f0497f48ec9464babfdbec23b6e236f'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:03:02.050624 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-04233f9f-844d-462d-941a-879038a2ac28 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '0f0497f48ec9464babfdbec23b6e236f'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:03:02.085927 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-449a8d30-6864-4627-be49-fe67a43132d1 admin admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:03:02.093488 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eaf8503b-23b7-416d-be2f-43ffa218cfe9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:03:02.094396 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaf8503b-23b7-416d-be2f-43ffa218cfe9 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:03:02.094671 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaf8503b-23b7-416d-be2f-43ffa218cfe9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:03:02.094934 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaf8503b-23b7-416d-be2f-43ffa218cfe9 admin admin] PATH_INFO: `/v3/users/88392549905845498b89f2d5d9f5f36b` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:03:02.096122 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eaf8503b-23b7-416d-be2f-43ffa218cfe9 admin admin] RBAC: Authorizing `identity:delete_user(user_id=88392549905845498b89f2d5d9f5f36b)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:03:02.098463 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eaf8503b-23b7-416d-be2f-43ffa218cfe9 admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:03:02.154248 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 994/1973] 10.222.0.27 () {58 vars in 1224 bytes} [Tue Aug 30 02:03:01 2022] DELETE /identity/v3/users/0f0497f48ec9464babfdbec23b6e236f => generated 0 bytes in 412 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 30 02:03:02.162901 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b84feed-9643-4ea8-88bd-bbe5ab318a89 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:03:02.162901 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b84feed-9643-4ea8-88bd-bbe5ab318a89 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:03:02.162901 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b84feed-9643-4ea8-88bd-bbe5ab318a89 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:03:02.179800 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-3b84feed-9643-4ea8-88bd-bbe5ab318a89 None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:03:02.181106 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-3b84feed-9643-4ea8-88bd-bbe5ab318a89 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:03:02.241371 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 995/1974] 10.222.0.27 () {58 vars in 958 bytes} [Tue Aug 30 02:03:02 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 02:03:02.252275 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b84feed-9643-4ea8-88bd-bbe5ab318a89 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:03:02.256611 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-3b84feed-9643-4ea8-88bd-bbe5ab318a89 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:03:02.319748 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b84feed-9643-4ea8-88bd-bbe5ab318a89 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:03:02.326010 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb0dc89c-d4ee-4125-b53f-da7d16754c77 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f3ea1580c56f4814a92bbb2acf542905', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4562f3951fbe465280073e60a11e44e2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:03:02.326723 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb0dc89c-d4ee-4125-b53f-da7d16754c77 service neutron] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:03:02.326920 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb0dc89c-d4ee-4125-b53f-da7d16754c77 service neutron] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:03:02.327213 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb0dc89c-d4ee-4125-b53f-da7d16754c77 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:03:02.337847 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb0dc89c-d4ee-4125-b53f-da7d16754c77 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:03:02.339652 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb0dc89c-d4ee-4125-b53f-da7d16754c77 service neutron] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:03:02.350234 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 996/1975] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Aug 30 02:03:02 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 98 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 02:03:02.374314 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-eaf8503b-23b7-416d-be2f-43ffa218cfe9 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '88392549905845498b89f2d5d9f5f36b'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:03:02.374990 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-eaf8503b-23b7-416d-be2f-43ffa218cfe9 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '88392549905845498b89f2d5d9f5f36b'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:03:02.393851 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:03:02.553274 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-eaf8503b-23b7-416d-be2f-43ffa218cfe9 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '88392549905845498b89f2d5d9f5f36b'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:03:02.554043 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-eaf8503b-23b7-416d-be2f-43ffa218cfe9 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '88392549905845498b89f2d5d9f5f36b'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:03:02.562276 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-eaf8503b-23b7-416d-be2f-43ffa218cfe9 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '88392549905845498b89f2d5d9f5f36b'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:03:02.579952 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 980/1976] 10.222.0.27 () {58 vars in 1224 bytes} [Tue Aug 30 02:03:02 2022] DELETE /identity/v3/users/88392549905845498b89f2d5d9f5f36b => generated 0 bytes in 578 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 30 02:03:02.873656 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:03:02.950998 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb0dc89c-d4ee-4125-b53f-da7d16754c77 service neutron] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:03:02.952659 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-fb0dc89c-d4ee-4125-b53f-da7d16754c77 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:03:02.966981 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb0dc89c-d4ee-4125-b53f-da7d16754c77 service neutron] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:03:03.033355 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c85efde-a4ed-4e4d-a036-6e65eefdf1c4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:03:03.034030 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c85efde-a4ed-4e4d-a036-6e65eefdf1c4 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:03:03.034313 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c85efde-a4ed-4e4d-a036-6e65eefdf1c4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:03:03.034586 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c85efde-a4ed-4e4d-a036-6e65eefdf1c4 admin admin] PATH_INFO: `/v3/projects/676b687ac2524ed7b343db1c39a3f985` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:03:03.035731 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c85efde-a4ed-4e4d-a036-6e65eefdf1c4 admin admin] RBAC: Authorizing `identity:delete_project(project_id=676b687ac2524ed7b343db1c39a3f985)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:03:03.037836 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c85efde-a4ed-4e4d-a036-6e65eefdf1c4 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:03:03.139023 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eaf8503b-23b7-416d-be2f-43ffa218cfe9 admin admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:03:03.139462 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-eaf8503b-23b7-416d-be2f-43ffa218cfe9 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:03:03.154712 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eaf8503b-23b7-416d-be2f-43ffa218cfe9 admin admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:03:03.165327 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6220a23-4e4c-4b92-a757-d7a3834f80d6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:03:03.166037 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6220a23-4e4c-4b92-a757-d7a3834f80d6 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:03:03.166037 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6220a23-4e4c-4b92-a757-d7a3834f80d6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:03:03.167273 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6220a23-4e4c-4b92-a757-d7a3834f80d6 admin admin] PATH_INFO: `/v3/projects/83602d77255e4422a6d3bcb34da64871` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:03:03.168233 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6220a23-4e4c-4b92-a757-d7a3834f80d6 admin admin] RBAC: Authorizing `identity:delete_project(project_id=83602d77255e4422a6d3bcb34da64871)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:03:03.170496 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6220a23-4e4c-4b92-a757-d7a3834f80d6 admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:03:03.223067 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-6c85efde-a4ed-4e4d-a036-6e65eefdf1c4 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '676b687ac2524ed7b343db1c39a3f985', 'request_id': 'req-6c85efde-a4ed-4e4d-a036-6e65eefdf1c4'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:03:03.229375 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-6c85efde-a4ed-4e4d-a036-6e65eefdf1c4 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '676b687ac2524ed7b343db1c39a3f985', 'request_id': 'req-6c85efde-a4ed-4e4d-a036-6e65eefdf1c4'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:03:03.235813 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-6c85efde-a4ed-4e4d-a036-6e65eefdf1c4 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '676b687ac2524ed7b343db1c39a3f985', 'request_id': 'req-6c85efde-a4ed-4e4d-a036-6e65eefdf1c4'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:03:03.491681 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-6c85efde-a4ed-4e4d-a036-6e65eefdf1c4 admin admin] The token cache is being invalidate because project 676b687ac2524ed7b343db1c39a3f985 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81577) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 30 02:03:03.491998 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-6c85efde-a4ed-4e4d-a036-6e65eefdf1c4 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:03:03.492616 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-6c85efde-a4ed-4e4d-a036-6e65eefdf1c4 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:03:03.494898 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 997/1977] 10.222.0.27 () {58 vars in 1233 bytes} [Tue Aug 30 02:03:02 2022] DELETE /identity/v3/projects/676b687ac2524ed7b343db1c39a3f985 => generated 0 bytes in 544 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 02:03:03.499837 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97e134b7-0ab4-4593-aade-7fed86eb7492 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:03:03.500131 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97e134b7-0ab4-4593-aade-7fed86eb7492 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:03:03.500458 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97e134b7-0ab4-4593-aade-7fed86eb7492 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:03:03.503177 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-d6220a23-4e4c-4b92-a757-d7a3834f80d6 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '83602d77255e4422a6d3bcb34da64871', 'request_id': 'req-d6220a23-4e4c-4b92-a757-d7a3834f80d6'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:03:03.508504 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-d6220a23-4e4c-4b92-a757-d7a3834f80d6 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '83602d77255e4422a6d3bcb34da64871', 'request_id': 'req-d6220a23-4e4c-4b92-a757-d7a3834f80d6'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:03:03.515418 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-d6220a23-4e4c-4b92-a757-d7a3834f80d6 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '83602d77255e4422a6d3bcb34da64871', 'request_id': 'req-d6220a23-4e4c-4b92-a757-d7a3834f80d6'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:03:03.520940 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-97e134b7-0ab4-4593-aade-7fed86eb7492 None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:03:03.522184 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-97e134b7-0ab4-4593-aade-7fed86eb7492 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:03:03.582399 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 998/1978] 10.222.0.27 () {58 vars in 958 bytes} [Tue Aug 30 02:03:03 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 30 02:03:03.589764 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ef72750-1403-4711-812c-b12f6ac202ef None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:03:03.590051 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ef72750-1403-4711-812c-b12f6ac202ef None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:03:03.590319 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ef72750-1403-4711-812c-b12f6ac202ef None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:03:03.610531 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-2ef72750-1403-4711-812c-b12f6ac202ef None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:03:03.611740 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-2ef72750-1403-4711-812c-b12f6ac202ef None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:03:03.619998 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 999/1979] 10.222.0.27 () {58 vars in 958 bytes} [Tue Aug 30 02:03:03 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 02:03:03.624081 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ef72750-1403-4711-812c-b12f6ac202ef None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:03:03.632871 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ef72750-1403-4711-812c-b12f6ac202ef None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:03:03.637620 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b9b7167-b5f6-40f6-808c-98fd6be5403b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:03:03.638196 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b9b7167-b5f6-40f6-808c-98fd6be5403b admin admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:03:03.638475 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b9b7167-b5f6-40f6-808c-98fd6be5403b admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:03:03.638748 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b9b7167-b5f6-40f6-808c-98fd6be5403b admin admin] PATH_INFO: `/v3/domains` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:03:03.639378 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b9b7167-b5f6-40f6-808c-98fd6be5403b admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 02:03:03.639682 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b9b7167-b5f6-40f6-808c-98fd6be5403b admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:03:03.641273 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b9b7167-b5f6-40f6-808c-98fd6be5403b admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:03:03.654982 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1000/1980] 10.222.0.27 () {58 vars in 1153 bytes} [Tue Aug 30 02:03:03 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 30 02:03:03.675597 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b9b7167-b5f6-40f6-808c-98fd6be5403b admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:03:03.686039 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b9b7167-b5f6-40f6-808c-98fd6be5403b admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:03:03.690681 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b569303-29e6-4551-a066-f2ac5241344c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:03:03.691279 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b569303-29e6-4551-a066-f2ac5241344c admin admin] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:03:03.691562 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b569303-29e6-4551-a066-f2ac5241344c admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:03:03.691839 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b569303-29e6-4551-a066-f2ac5241344c admin admin] PATH_INFO: `/v3/projects` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:03:03.692531 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b569303-29e6-4551-a066-f2ac5241344c admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:03:03.694493 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b569303-29e6-4551-a066-f2ac5241344c admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:03:03.718681 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-d6220a23-4e4c-4b92-a757-d7a3834f80d6 admin admin] The token cache is being invalidate because project 83602d77255e4422a6d3bcb34da64871 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81578) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 30 02:03:03.718970 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-d6220a23-4e4c-4b92-a757-d7a3834f80d6 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:03:03.719601 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-d6220a23-4e4c-4b92-a757-d7a3834f80d6 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:03:03.721231 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 981/1981] 10.222.0.27 () {58 vars in 1233 bytes} [Tue Aug 30 02:03:03 2022] DELETE /identity/v3/projects/83602d77255e4422a6d3bcb34da64871 => generated 0 bytes in 584 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 02:03:03.733148 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcc4760a-1054-4028-9893-7344f788c1a8 None None] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:03:03.733439 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcc4760a-1054-4028-9893-7344f788c1a8 None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:03:03.733754 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcc4760a-1054-4028-9893-7344f788c1a8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:03:03.753981 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.auth.core [None req-bcc4760a-1054-4028-9893-7344f788c1a8 None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81578) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:03:03.755207 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-bcc4760a-1054-4028-9893-7344f788c1a8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:03:03.763712 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 982/1982] 10.222.0.27 () {58 vars in 958 bytes} [Tue Aug 30 02:03:03 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 30 02:03:03.772598 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5453f421-3bc2-41bf-bcaa-47dd1c7cfd65 None None] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:03:03.772883 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5453f421-3bc2-41bf-bcaa-47dd1c7cfd65 None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:03:03.773183 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5453f421-3bc2-41bf-bcaa-47dd1c7cfd65 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:03:03.793273 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.auth.core [None req-5453f421-3bc2-41bf-bcaa-47dd1c7cfd65 None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81578) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:03:03.794481 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-5453f421-3bc2-41bf-bcaa-47dd1c7cfd65 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:03:03.802568 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 983/1983] 10.222.0.27 () {58 vars in 958 bytes} [Tue Aug 30 02:03:03 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 30 02:03:03.806438 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5453f421-3bc2-41bf-bcaa-47dd1c7cfd65 None None] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:03:03.816053 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5453f421-3bc2-41bf-bcaa-47dd1c7cfd65 None None] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:03:03.820773 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0055363d-28e2-420e-bde0-4587aeca0a3f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:03:03.826971 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0055363d-28e2-420e-bde0-4587aeca0a3f admin admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:03:03.826971 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0055363d-28e2-420e-bde0-4587aeca0a3f admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:03:03.826971 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0055363d-28e2-420e-bde0-4587aeca0a3f admin admin] PATH_INFO: `/v3/domains` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:03:03.826971 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0055363d-28e2-420e-bde0-4587aeca0a3f admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81578) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 02:03:03.826971 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0055363d-28e2-420e-bde0-4587aeca0a3f admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:03:03.826971 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0055363d-28e2-420e-bde0-4587aeca0a3f admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:03:03.836040 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 984/1984] 10.222.0.27 () {58 vars in 1153 bytes} [Tue Aug 30 02:03:03 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 30 02:03:03.840364 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0055363d-28e2-420e-bde0-4587aeca0a3f admin admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:03:03.850459 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0055363d-28e2-420e-bde0-4587aeca0a3f admin admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:03:03.859019 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5c966b8-d7e3-4e39-9083-d7aa63e3a6a1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:03:03.859019 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5c966b8-d7e3-4e39-9083-d7aa63e3a6a1 admin admin] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:03:03.859019 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5c966b8-d7e3-4e39-9083-d7aa63e3a6a1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:03:03.859019 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5c966b8-d7e3-4e39-9083-d7aa63e3a6a1 admin admin] PATH_INFO: `/v3/projects` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:03:03.859311 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5c966b8-d7e3-4e39-9083-d7aa63e3a6a1 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:03:03.861158 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5c966b8-d7e3-4e39-9083-d7aa63e3a6a1 admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:03:03.886778 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1001/1985] 10.222.0.27 () {60 vars in 1153 bytes} [Tue Aug 30 02:03:03 2022] POST /identity/v3/projects => generated 371 bytes in 212 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 30 02:03:03.891258 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b569303-29e6-4551-a066-f2ac5241344c admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:03:03.893111 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-8b569303-29e6-4551-a066-f2ac5241344c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:03:03.956565 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b569303-29e6-4551-a066-f2ac5241344c admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:03:03.961827 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcb335be-6b3d-4d65-9f74-c6edc3b6b3af admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:03:03.962451 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcb335be-6b3d-4d65-9f74-c6edc3b6b3af admin admin] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:03:03.962733 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcb335be-6b3d-4d65-9f74-c6edc3b6b3af admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:03:03.963554 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcb335be-6b3d-4d65-9f74-c6edc3b6b3af admin admin] PATH_INFO: `/v3/users` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:03:03.964276 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bcb335be-6b3d-4d65-9f74-c6edc3b6b3af admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:03:03.966236 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bcb335be-6b3d-4d65-9f74-c6edc3b6b3af admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:03:04.052658 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 985/1986] 10.222.0.27 () {60 vars in 1153 bytes} [Tue Aug 30 02:03:03 2022] POST /identity/v3/projects => generated 365 bytes in 213 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 30 02:03:04.056869 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5c966b8-d7e3-4e39-9083-d7aa63e3a6a1 admin admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:03:04.092271 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1002/1987] 10.222.0.27 () {60 vars in 1144 bytes} [Tue Aug 30 02:03:03 2022] POST /identity/v3/users => generated 286 bytes in 202 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 30 02:03:04.096268 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcb335be-6b3d-4d65-9f74-c6edc3b6b3af admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:03:04.106760 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcb335be-6b3d-4d65-9f74-c6edc3b6b3af admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:03:04.122767 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5c966b8-d7e3-4e39-9083-d7aa63e3a6a1 admin admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:03:04.127456 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10a1795d-3491-4842-884c-dd9fc34ace0f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:03:04.128060 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10a1795d-3491-4842-884c-dd9fc34ace0f admin admin] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:03:04.128341 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10a1795d-3491-4842-884c-dd9fc34ace0f admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:03:04.128611 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10a1795d-3491-4842-884c-dd9fc34ace0f admin admin] PATH_INFO: `/v3/users` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:03:04.129239 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10a1795d-3491-4842-884c-dd9fc34ace0f admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:03:04.131196 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10a1795d-3491-4842-884c-dd9fc34ace0f admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:03:04.163644 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6389c911-cdd7-47a8-b031-4b2524db64eb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:03:04.164169 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6389c911-cdd7-47a8-b031-4b2524db64eb admin admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:03:04.164447 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6389c911-cdd7-47a8-b031-4b2524db64eb admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:03:04.164717 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6389c911-cdd7-47a8-b031-4b2524db64eb admin admin] PATH_INFO: `/v3/roles` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:03:04.165363 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6389c911-cdd7-47a8-b031-4b2524db64eb admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:03:04.167573 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6389c911-cdd7-47a8-b031-4b2524db64eb admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:03:04.177842 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1003/1988] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Aug 30 02:03:04 2022] GET /identity/v3/roles => generated 1420 bytes in 82 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 30 02:03:04.181719 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6389c911-cdd7-47a8-b031-4b2524db64eb admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:03:04.193118 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6389c911-cdd7-47a8-b031-4b2524db64eb admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:03:04.202518 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4eccf93b-a8ac-495b-9073-bc27889ba9ac admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:03:04.208730 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4eccf93b-a8ac-495b-9073-bc27889ba9ac admin admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:03:04.208730 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4eccf93b-a8ac-495b-9073-bc27889ba9ac admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:03:04.208730 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4eccf93b-a8ac-495b-9073-bc27889ba9ac admin admin] PATH_INFO: `/v3/roles` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:03:04.208730 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4eccf93b-a8ac-495b-9073-bc27889ba9ac admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:03:04.208730 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4eccf93b-a8ac-495b-9073-bc27889ba9ac admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:03:04.217435 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1004/1989] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Aug 30 02:03:04 2022] GET /identity/v3/roles => generated 1420 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 30 02:03:04.221563 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4eccf93b-a8ac-495b-9073-bc27889ba9ac admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:03:04.231774 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4eccf93b-a8ac-495b-9073-bc27889ba9ac admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:03:04.236350 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1525eea-6b0c-4249-aa94-fd75bcdd0e97 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:03:04.236983 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1525eea-6b0c-4249-aa94-fd75bcdd0e97 admin admin] REQUEST_METHOD: `PUT` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:03:04.242611 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1525eea-6b0c-4249-aa94-fd75bcdd0e97 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:03:04.242611 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1525eea-6b0c-4249-aa94-fd75bcdd0e97 admin admin] PATH_INFO: `/v3/projects/8527acd3c1964f7f9d67b413bf9ef488/users/6f1e736fab4f47ac868d1bed7400c571/roles/1fb81afcb9f245f3b989add94598a063` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:03:04.251979 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1525eea-6b0c-4249-aa94-fd75bcdd0e97 admin admin] RBAC: Authorizing `identity:create_grant(project_id=8527acd3c1964f7f9d67b413bf9ef488, user_id=6f1e736fab4f47ac868d1bed7400c571, role_id=1fb81afcb9f245f3b989add94598a063)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:03:04.254733 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1525eea-6b0c-4249-aa94-fd75bcdd0e97 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:03:04.279730 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 986/1990] 10.222.0.27 () {60 vars in 1144 bytes} [Tue Aug 30 02:03:04 2022] POST /identity/v3/users => generated 283 bytes in 223 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 30 02:03:04.284789 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10a1795d-3491-4842-884c-dd9fc34ace0f admin admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:03:04.296234 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10a1795d-3491-4842-884c-dd9fc34ace0f admin admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:03:04.302412 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c93a910-06ae-4d81-894b-52907e98a27b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:03:04.303151 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c93a910-06ae-4d81-894b-52907e98a27b admin admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:03:04.303428 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c93a910-06ae-4d81-894b-52907e98a27b admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:03:04.303699 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c93a910-06ae-4d81-894b-52907e98a27b admin admin] PATH_INFO: `/v3/roles` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:03:04.304340 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c93a910-06ae-4d81-894b-52907e98a27b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:03:04.306265 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c93a910-06ae-4d81-894b-52907e98a27b admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:03:04.319777 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 987/1991] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Aug 30 02:03:04 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 30 02:03:04.322817 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c93a910-06ae-4d81-894b-52907e98a27b admin admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:03:04.334414 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c93a910-06ae-4d81-894b-52907e98a27b admin admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:03:04.340338 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5a42b32-43cb-402b-be8f-a23e1f15d974 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:03:04.340960 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5a42b32-43cb-402b-be8f-a23e1f15d974 admin admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:03:04.341279 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5a42b32-43cb-402b-be8f-a23e1f15d974 admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:03:04.342007 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5a42b32-43cb-402b-be8f-a23e1f15d974 admin admin] PATH_INFO: `/v3/roles` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:03:04.342662 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5a42b32-43cb-402b-be8f-a23e1f15d974 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:03:04.344297 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5a42b32-43cb-402b-be8f-a23e1f15d974 admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:03:04.359624 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 988/1992] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Aug 30 02:03:04 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 30 02:03:04.360874 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1005/1993] 10.222.0.27 () {60 vars in 1483 bytes} [Tue Aug 30 02:03:04 2022] PUT /identity/v3/projects/8527acd3c1964f7f9d67b413bf9ef488/users/6f1e736fab4f47ac868d1bed7400c571/roles/1fb81afcb9f245f3b989add94598a063 => generated 0 bytes in 140 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 30 02:03:04.364678 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5a42b32-43cb-402b-be8f-a23e1f15d974 admin admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:03:04.372180 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c618afb5-eab0-45c0-afa9-37ac9cd23853 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:03:04.372490 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c618afb5-eab0-45c0-afa9-37ac9cd23853 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:03:04.372767 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c618afb5-eab0-45c0-afa9-37ac9cd23853 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:03:04.375325 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5a42b32-43cb-402b-be8f-a23e1f15d974 admin admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:03:04.408429 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-c618afb5-eab0-45c0-afa9-37ac9cd23853 None None] MFA Rules not processed for user `6f1e736fab4f47ac868d1bed7400c571`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:03:04.410012 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-c618afb5-eab0-45c0-afa9-37ac9cd23853 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:03:04.470660 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=86,keystone:DELETE=10,keystone:INSERT=4 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:03:04.475750 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9768f62c-005f-4008-9709-a7fd4c792a0a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:03:04.476671 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9768f62c-005f-4008-9709-a7fd4c792a0a admin admin] REQUEST_METHOD: `PUT` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:03:04.477219 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9768f62c-005f-4008-9709-a7fd4c792a0a admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:03:04.477890 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9768f62c-005f-4008-9709-a7fd4c792a0a admin admin] PATH_INFO: `/v3/projects/e0bb2b1a596c4e33ba7ab6d57f218e83/users/3d9a18e5feca444287b8f01890121f38/roles/1fb81afcb9f245f3b989add94598a063` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:03:04.493831 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1006/1994] 10.222.0.27 () {58 vars in 958 bytes} [Tue Aug 30 02:03:04 2022] POST /identity/v3/auth/tokens => generated 3248 bytes in 123 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 02:03:04.500394 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9768f62c-005f-4008-9709-a7fd4c792a0a admin admin] RBAC: Authorizing `identity:create_grant(project_id=e0bb2b1a596c4e33ba7ab6d57f218e83, user_id=3d9a18e5feca444287b8f01890121f38, role_id=1fb81afcb9f245f3b989add94598a063)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:03:04.503108 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9768f62c-005f-4008-9709-a7fd4c792a0a admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:03:04.515228 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c618afb5-eab0-45c0-afa9-37ac9cd23853 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:03:04.515228 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-c618afb5-eab0-45c0-afa9-37ac9cd23853 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:03:04.575191 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c618afb5-eab0-45c0-afa9-37ac9cd23853 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:03:04.579110 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-851213ce-c50e-4879-be0b-848f31fb9b39 service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3caff62093d748f48596574836e12e42', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4562f3951fbe465280073e60a11e44e2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:03:04.579732 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-851213ce-c50e-4879-be0b-848f31fb9b39 service glance] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:03:04.580007 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-851213ce-c50e-4879-be0b-848f31fb9b39 service glance] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:03:04.580350 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-851213ce-c50e-4879-be0b-848f31fb9b39 service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:03:04.590792 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-851213ce-c50e-4879-be0b-848f31fb9b39 service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:03:04.592494 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-851213ce-c50e-4879-be0b-848f31fb9b39 service glance] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:03:04.594657 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 989/1995] 10.222.0.27 () {60 vars in 1483 bytes} [Tue Aug 30 02:03:04 2022] PUT /identity/v3/projects/e0bb2b1a596c4e33ba7ab6d57f218e83/users/3d9a18e5feca444287b8f01890121f38/roles/1fb81afcb9f245f3b989add94598a063 => generated 0 bytes in 230 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 02:03:04.605712 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1007/1996] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Aug 30 02:03:04 2022] GET /identity/v3/auth/tokens => generated 3248 bytes in 100 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 02:03:04.605890 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13ff542c-eb0d-4cc8-8ce8-e7c126640382 None None] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:03:04.606174 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13ff542c-eb0d-4cc8-8ce8-e7c126640382 None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:03:04.606448 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13ff542c-eb0d-4cc8-8ce8-e7c126640382 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:03:04.645506 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.auth.core [None req-13ff542c-eb0d-4cc8-8ce8-e7c126640382 None None] MFA Rules not processed for user `3d9a18e5feca444287b8f01890121f38`. Rule list: `[]` (Enabled: `True`). {{(pid=81578) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:03:04.646798 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-13ff542c-eb0d-4cc8-8ce8-e7c126640382 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:03:04.725014 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 990/1997] 10.222.0.27 () {58 vars in 958 bytes} [Tue Aug 30 02:03:04 2022] POST /identity/v3/auth/tokens => generated 3242 bytes in 120 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 02:03:04.729493 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-851213ce-c50e-4879-be0b-848f31fb9b39 service glance] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:03:04.740936 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-851213ce-c50e-4879-be0b-848f31fb9b39 service glance] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:03:04.816707 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41d2f1aa-0b4c-41ff-b152-7d927466385a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:03:04.817438 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41d2f1aa-0b4c-41ff-b152-7d927466385a admin admin] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:03:04.817836 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41d2f1aa-0b4c-41ff-b152-7d927466385a admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:03:04.818199 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41d2f1aa-0b4c-41ff-b152-7d927466385a admin admin] PATH_INFO: `/v3/projects` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:03:04.819108 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41d2f1aa-0b4c-41ff-b152-7d927466385a admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:03:04.821475 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41d2f1aa-0b4c-41ff-b152-7d927466385a admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:03:04.956261 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=88,keystone:DELETE=10,keystone:INSERT=2 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:03:04.966998 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1008/1998] 10.222.0.27 () {60 vars in 1153 bytes} [Tue Aug 30 02:03:04 2022] POST /identity/v3/projects => generated 365 bytes in 235 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 02:03:04.968392 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13ff542c-eb0d-4cc8-8ce8-e7c126640382 None None] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:03:04.980009 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13ff542c-eb0d-4cc8-8ce8-e7c126640382 None None] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:03:05.051692 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12b6b490-992c-446d-bd55-fe68953ca378 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:03:05.052306 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12b6b490-992c-446d-bd55-fe68953ca378 admin admin] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:03:05.052589 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12b6b490-992c-446d-bd55-fe68953ca378 admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:03:05.052861 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12b6b490-992c-446d-bd55-fe68953ca378 admin admin] PATH_INFO: `/v3/users` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:03:05.053566 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12b6b490-992c-446d-bd55-fe68953ca378 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:03:05.055586 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12b6b490-992c-446d-bd55-fe68953ca378 admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:03:05.205340 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 991/1999] 10.222.0.27 () {60 vars in 1144 bytes} [Tue Aug 30 02:03:04 2022] POST /identity/v3/users => generated 289 bytes in 238 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 30 02:03:05.209316 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41d2f1aa-0b4c-41ff-b152-7d927466385a admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:03:05.221716 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41d2f1aa-0b4c-41ff-b152-7d927466385a admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:03:05.226461 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1db53305-5594-46e6-b3e8-69c291ae1610 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:03:05.227072 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1db53305-5594-46e6-b3e8-69c291ae1610 admin admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:03:05.227352 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1db53305-5594-46e6-b3e8-69c291ae1610 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:03:05.227623 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1db53305-5594-46e6-b3e8-69c291ae1610 admin admin] PATH_INFO: `/v3/roles` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:03:05.228294 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1db53305-5594-46e6-b3e8-69c291ae1610 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:03:05.230622 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1db53305-5594-46e6-b3e8-69c291ae1610 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:03:05.242376 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1009/2000] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Aug 30 02:03:05 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 02:03:05.247894 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12b6b490-992c-446d-bd55-fe68953ca378 admin admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:03:05.258033 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12b6b490-992c-446d-bd55-fe68953ca378 admin admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:03:05.262995 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-166f3bb4-d5a1-4d21-81b8-3cb1dfe6ae35 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:03:05.263683 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-166f3bb4-d5a1-4d21-81b8-3cb1dfe6ae35 admin admin] REQUEST_METHOD: `PUT` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:03:05.263964 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-166f3bb4-d5a1-4d21-81b8-3cb1dfe6ae35 admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:03:05.264232 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-166f3bb4-d5a1-4d21-81b8-3cb1dfe6ae35 admin admin] PATH_INFO: `/v3/projects/98da7ed713a147bfbe5aa58a2e59447b/users/7a9e18a244f446a2af2eab3110cefe8d/roles/ead5fac86a0747ea8823a82efcea8ed7` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:03:05.280248 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-166f3bb4-d5a1-4d21-81b8-3cb1dfe6ae35 admin admin] RBAC: Authorizing `identity:create_grant(project_id=98da7ed713a147bfbe5aa58a2e59447b, user_id=7a9e18a244f446a2af2eab3110cefe8d, role_id=ead5fac86a0747ea8823a82efcea8ed7)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:03:05.282513 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-166f3bb4-d5a1-4d21-81b8-3cb1dfe6ae35 admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:03:05.468769 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 992/2001] 10.222.0.27 () {60 vars in 1483 bytes} [Tue Aug 30 02:03:05 2022] PUT /identity/v3/projects/98da7ed713a147bfbe5aa58a2e59447b/users/7a9e18a244f446a2af2eab3110cefe8d/roles/ead5fac86a0747ea8823a82efcea8ed7 => generated 0 bytes in 221 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 02:03:05.472909 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1db53305-5594-46e6-b3e8-69c291ae1610 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:03:05.487189 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1db53305-5594-46e6-b3e8-69c291ae1610 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:03:05.546111 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef3ae0bb-d321-4bf0-80c1-1c2a5b83e675 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:03:05.546848 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef3ae0bb-d321-4bf0-80c1-1c2a5b83e675 admin admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:03:05.547124 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef3ae0bb-d321-4bf0-80c1-1c2a5b83e675 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:03:05.547391 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef3ae0bb-d321-4bf0-80c1-1c2a5b83e675 admin admin] PATH_INFO: `/v3/roles` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:03:05.548022 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef3ae0bb-d321-4bf0-80c1-1c2a5b83e675 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:03:05.549637 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef3ae0bb-d321-4bf0-80c1-1c2a5b83e675 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:03:05.559860 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1010/2002] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Aug 30 02:03:05 2022] GET /identity/v3/roles => generated 1420 bytes in 87 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 02:03:05.564288 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-166f3bb4-d5a1-4d21-81b8-3cb1dfe6ae35 admin admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:03:05.576356 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-166f3bb4-d5a1-4d21-81b8-3cb1dfe6ae35 admin admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:03:05.584896 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff4924d5-5dfe-4815-b238-90e8276e3aa3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:03:05.585628 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff4924d5-5dfe-4815-b238-90e8276e3aa3 admin admin] REQUEST_METHOD: `PUT` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:03:05.585908 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff4924d5-5dfe-4815-b238-90e8276e3aa3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:03:05.586179 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff4924d5-5dfe-4815-b238-90e8276e3aa3 admin admin] PATH_INFO: `/v3/projects/98da7ed713a147bfbe5aa58a2e59447b/users/7a9e18a244f446a2af2eab3110cefe8d/roles/ead5fac86a0747ea8823a82efcea8ed7` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:03:05.588325 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff4924d5-5dfe-4815-b238-90e8276e3aa3 admin admin] RBAC: Authorizing `identity:create_grant(project_id=98da7ed713a147bfbe5aa58a2e59447b, user_id=7a9e18a244f446a2af2eab3110cefe8d, role_id=ead5fac86a0747ea8823a82efcea8ed7)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:03:05.590996 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff4924d5-5dfe-4815-b238-90e8276e3aa3 admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:03:05.601973 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 993/2003] 10.222.0.27 () {60 vars in 1483 bytes} [Tue Aug 30 02:03:05 2022] PUT /identity/v3/projects/98da7ed713a147bfbe5aa58a2e59447b/users/7a9e18a244f446a2af2eab3110cefe8d/roles/ead5fac86a0747ea8823a82efcea8ed7 => generated 0 bytes in 38 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 02:03:05.613010 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7363d368-dffc-449c-b6ba-9c54f321c8a2 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:03:05.613298 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7363d368-dffc-449c-b6ba-9c54f321c8a2 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:03:05.613613 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7363d368-dffc-449c-b6ba-9c54f321c8a2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:03:05.643933 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-7363d368-dffc-449c-b6ba-9c54f321c8a2 None None] MFA Rules not processed for user `7a9e18a244f446a2af2eab3110cefe8d`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:03:05.645143 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-7363d368-dffc-449c-b6ba-9c54f321c8a2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:03:05.720539 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1011/2004] 10.222.0.27 () {58 vars in 958 bytes} [Tue Aug 30 02:03:05 2022] POST /identity/v3/auth/tokens => generated 3309 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 02:03:05.890797 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff4924d5-5dfe-4815-b238-90e8276e3aa3 admin admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:03:05.896023 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-ff4924d5-5dfe-4815-b238-90e8276e3aa3 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:03:05.973627 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff4924d5-5dfe-4815-b238-90e8276e3aa3 admin admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:03:05.977108 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6dc9da8c-7d8f-47f1-a326-ba694c7cd8d7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:03:05.977670 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dc9da8c-7d8f-47f1-a326-ba694c7cd8d7 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:03:05.977950 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dc9da8c-7d8f-47f1-a326-ba694c7cd8d7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:03:05.978222 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dc9da8c-7d8f-47f1-a326-ba694c7cd8d7 admin admin] PATH_INFO: `/v3/users/5bc0855d88664e548a50bc6305dfdb49` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:03:05.980046 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6dc9da8c-7d8f-47f1-a326-ba694c7cd8d7 admin admin] RBAC: Authorizing `identity:delete_user(user_id=5bc0855d88664e548a50bc6305dfdb49)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:03:05.982079 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6dc9da8c-7d8f-47f1-a326-ba694c7cd8d7 admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:03:06.018532 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7363d368-dffc-449c-b6ba-9c54f321c8a2 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:03:06.020121 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-7363d368-dffc-449c-b6ba-9c54f321c8a2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:03:06.087252 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7363d368-dffc-449c-b6ba-9c54f321c8a2 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:03:06.090942 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e7d5ff4-0556-4276-aaef-9d38c449a052 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ab936b3767a8417aa80c8c8f1b84b667', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4562f3951fbe465280073e60a11e44e2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:03:06.091558 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e7d5ff4-0556-4276-aaef-9d38c449a052 service cinder] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:03:06.092093 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e7d5ff4-0556-4276-aaef-9d38c449a052 service cinder] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:03:06.092364 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e7d5ff4-0556-4276-aaef-9d38c449a052 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:03:06.101625 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e7d5ff4-0556-4276-aaef-9d38c449a052 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:03:06.103065 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e7d5ff4-0556-4276-aaef-9d38c449a052 service cinder] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:03:06.113054 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1012/2005] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Aug 30 02:03:06 2022] GET /identity/v3/auth/tokens => generated 3309 bytes in 95 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 30 02:03:06.214630 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-6dc9da8c-7d8f-47f1-a326-ba694c7cd8d7 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '5bc0855d88664e548a50bc6305dfdb49'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:03:06.215301 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-6dc9da8c-7d8f-47f1-a326-ba694c7cd8d7 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '5bc0855d88664e548a50bc6305dfdb49'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:03:06.364794 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-6dc9da8c-7d8f-47f1-a326-ba694c7cd8d7 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '5bc0855d88664e548a50bc6305dfdb49'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:03:06.365385 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-6dc9da8c-7d8f-47f1-a326-ba694c7cd8d7 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '5bc0855d88664e548a50bc6305dfdb49'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:03:06.373052 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-6dc9da8c-7d8f-47f1-a326-ba694c7cd8d7 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '5bc0855d88664e548a50bc6305dfdb49'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:03:06.394544 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 994/2006] 10.222.0.27 () {58 vars in 1224 bytes} [Tue Aug 30 02:03:05 2022] DELETE /identity/v3/users/5bc0855d88664e548a50bc6305dfdb49 => generated 0 bytes in 504 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 02:03:06.399051 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-224307f0-2ffb-452f-81f8-6031744c76ea None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:03:06.399222 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-224307f0-2ffb-452f-81f8-6031744c76ea None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:03:06.399552 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-224307f0-2ffb-452f-81f8-6031744c76ea None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:03:06.404927 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6dc9da8c-7d8f-47f1-a326-ba694c7cd8d7 admin admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:03:06.406305 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-6dc9da8c-7d8f-47f1-a326-ba694c7cd8d7 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:03:06.420256 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-224307f0-2ffb-452f-81f8-6031744c76ea None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:03:06.421443 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-224307f0-2ffb-452f-81f8-6031744c76ea None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:03:06.465336 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6dc9da8c-7d8f-47f1-a326-ba694c7cd8d7 admin admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:03:06.469272 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-842d8a90-9c5a-493b-a8ac-03c9ebec3bce service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ab936b3767a8417aa80c8c8f1b84b667', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4562f3951fbe465280073e60a11e44e2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:03:06.469911 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-842d8a90-9c5a-493b-a8ac-03c9ebec3bce service cinder] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:03:06.470190 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-842d8a90-9c5a-493b-a8ac-03c9ebec3bce service cinder] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:03:06.470463 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-842d8a90-9c5a-493b-a8ac-03c9ebec3bce service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:03:06.472282 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-842d8a90-9c5a-493b-a8ac-03c9ebec3bce service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:03:06.487644 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1013/2007] 10.222.0.27 () {58 vars in 958 bytes} [Tue Aug 30 02:03:06 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 02:03:06.499590 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-224307f0-2ffb-452f-81f8-6031744c76ea None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:03:06.501116 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-224307f0-2ffb-452f-81f8-6031744c76ea None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:03:06.549612 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-842d8a90-9c5a-493b-a8ac-03c9ebec3bce service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:03:06.551135 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-842d8a90-9c5a-493b-a8ac-03c9ebec3bce service cinder] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:03:06.562263 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 995/2008] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Aug 30 02:03:06 2022] GET /identity/v3/auth/tokens => generated 3242 bytes in 157 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 30 02:03:06.562763 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-224307f0-2ffb-452f-81f8-6031744c76ea None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:03:06.566942 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0031f789-232d-4f4d-9cc5-b0c93f9f8dcf service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f3ea1580c56f4814a92bbb2acf542905', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4562f3951fbe465280073e60a11e44e2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:03:06.567569 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0031f789-232d-4f4d-9cc5-b0c93f9f8dcf service neutron] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:03:06.567845 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0031f789-232d-4f4d-9cc5-b0c93f9f8dcf service neutron] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:03:06.568113 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0031f789-232d-4f4d-9cc5-b0c93f9f8dcf service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:03:06.580346 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0031f789-232d-4f4d-9cc5-b0c93f9f8dcf service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:03:06.591033 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0031f789-232d-4f4d-9cc5-b0c93f9f8dcf service neutron] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:03:06.592256 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1014/2009] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Aug 30 02:03:06 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 93 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 30 02:03:07.155699 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-842d8a90-9c5a-493b-a8ac-03c9ebec3bce service cinder] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:03:07.157149 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-842d8a90-9c5a-493b-a8ac-03c9ebec3bce service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:03:07.171126 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-842d8a90-9c5a-493b-a8ac-03c9ebec3bce service cinder] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:03:07.177642 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b96a36c-5f02-4d66-b445-ed9149ea36cc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:03:07.178403 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b96a36c-5f02-4d66-b445-ed9149ea36cc admin admin] REQUEST_METHOD: `DELETE` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:03:07.178785 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b96a36c-5f02-4d66-b445-ed9149ea36cc admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:03:07.179208 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b96a36c-5f02-4d66-b445-ed9149ea36cc admin admin] PATH_INFO: `/v3/projects/476f9e3d07ff4b688527b640a0ad7256` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:03:07.180449 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b96a36c-5f02-4d66-b445-ed9149ea36cc admin admin] RBAC: Authorizing `identity:delete_project(project_id=476f9e3d07ff4b688527b640a0ad7256)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:03:07.183096 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b96a36c-5f02-4d66-b445-ed9149ea36cc admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:03:07.194674 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:03:07.374626 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-0b96a36c-5f02-4d66-b445-ed9149ea36cc admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '476f9e3d07ff4b688527b640a0ad7256', 'request_id': 'req-0b96a36c-5f02-4d66-b445-ed9149ea36cc'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:03:07.379948 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-0b96a36c-5f02-4d66-b445-ed9149ea36cc admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '476f9e3d07ff4b688527b640a0ad7256', 'request_id': 'req-0b96a36c-5f02-4d66-b445-ed9149ea36cc'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:03:07.386016 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-0b96a36c-5f02-4d66-b445-ed9149ea36cc admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '476f9e3d07ff4b688527b640a0ad7256', 'request_id': 'req-0b96a36c-5f02-4d66-b445-ed9149ea36cc'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:03:07.543909 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-0b96a36c-5f02-4d66-b445-ed9149ea36cc admin admin] The token cache is being invalidate because project 476f9e3d07ff4b688527b640a0ad7256 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81578) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 30 02:03:07.544250 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-0b96a36c-5f02-4d66-b445-ed9149ea36cc admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:03:07.545047 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-0b96a36c-5f02-4d66-b445-ed9149ea36cc admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:03:07.546839 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 996/2010] 10.222.0.27 () {58 vars in 1233 bytes} [Tue Aug 30 02:03:07 2022] DELETE /identity/v3/projects/476f9e3d07ff4b688527b640a0ad7256 => generated 0 bytes in 392 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 02:03:07.550932 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0031f789-232d-4f4d-9cc5-b0c93f9f8dcf service neutron] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:03:07.561451 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0031f789-232d-4f4d-9cc5-b0c93f9f8dcf service neutron] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:03:07.626677 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2e514bd-20cb-4d24-9c80-a2763337e836 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:03:07.627346 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2e514bd-20cb-4d24-9c80-a2763337e836 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:03:07.627624 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2e514bd-20cb-4d24-9c80-a2763337e836 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:03:07.627920 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2e514bd-20cb-4d24-9c80-a2763337e836 admin admin] PATH_INFO: `/v3/users/1a43058cf7e54449828a43731fa71d6a` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:03:07.629026 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2e514bd-20cb-4d24-9c80-a2763337e836 admin admin] RBAC: Authorizing `identity:delete_user(user_id=1a43058cf7e54449828a43731fa71d6a)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:03:07.631106 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2e514bd-20cb-4d24-9c80-a2763337e836 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:03:07.759328 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:03:07.847964 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-e2e514bd-20cb-4d24-9c80-a2763337e836 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '1a43058cf7e54449828a43731fa71d6a'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:03:07.848631 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-e2e514bd-20cb-4d24-9c80-a2763337e836 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '1a43058cf7e54449828a43731fa71d6a'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:03:07.849239 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-e2e514bd-20cb-4d24-9c80-a2763337e836 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '1a43058cf7e54449828a43731fa71d6a'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:03:07.863504 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-e2e514bd-20cb-4d24-9c80-a2763337e836 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '1a43058cf7e54449828a43731fa71d6a'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:03:07.871404 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-e2e514bd-20cb-4d24-9c80-a2763337e836 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '1a43058cf7e54449828a43731fa71d6a'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:03:07.953875 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1015/2011] 10.222.0.27 () {58 vars in 1224 bytes} [Tue Aug 30 02:03:07 2022] DELETE /identity/v3/users/1a43058cf7e54449828a43731fa71d6a => generated 0 bytes in 403 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 02:03:08.508481 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b96a36c-5f02-4d66-b445-ed9149ea36cc admin admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:03:08.510474 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-0b96a36c-5f02-4d66-b445-ed9149ea36cc admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:03:08.527579 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b96a36c-5f02-4d66-b445-ed9149ea36cc admin admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:03:08.594870 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-907b45b1-8e84-440c-b91e-e98c8be0373d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:03:08.595500 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-907b45b1-8e84-440c-b91e-e98c8be0373d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:03:08.595780 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-907b45b1-8e84-440c-b91e-e98c8be0373d admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:03:08.596049 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-907b45b1-8e84-440c-b91e-e98c8be0373d admin admin] PATH_INFO: `/v3/projects/a34963efe84f4775a418fdc92c100606` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:03:08.597157 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-907b45b1-8e84-440c-b91e-e98c8be0373d admin admin] RBAC: Authorizing `identity:delete_project(project_id=a34963efe84f4775a418fdc92c100606)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:03:08.600301 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-907b45b1-8e84-440c-b91e-e98c8be0373d admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:03:08.720460 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-907b45b1-8e84-440c-b91e-e98c8be0373d admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'a34963efe84f4775a418fdc92c100606', 'request_id': 'req-907b45b1-8e84-440c-b91e-e98c8be0373d'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:03:08.727460 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-907b45b1-8e84-440c-b91e-e98c8be0373d admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'a34963efe84f4775a418fdc92c100606', 'request_id': 'req-907b45b1-8e84-440c-b91e-e98c8be0373d'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:03:08.734033 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-907b45b1-8e84-440c-b91e-e98c8be0373d admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'a34963efe84f4775a418fdc92c100606', 'request_id': 'req-907b45b1-8e84-440c-b91e-e98c8be0373d'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:03:08.808373 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-907b45b1-8e84-440c-b91e-e98c8be0373d admin admin] The token cache is being invalidate because project a34963efe84f4775a418fdc92c100606 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81578) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 30 02:03:08.808810 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-907b45b1-8e84-440c-b91e-e98c8be0373d admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:03:08.809635 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-907b45b1-8e84-440c-b91e-e98c8be0373d admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:03:08.811520 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 997/2012] 10.222.0.27 () {58 vars in 1233 bytes} [Tue Aug 30 02:03:08 2022] DELETE /identity/v3/projects/a34963efe84f4775a418fdc92c100606 => generated 0 bytes in 303 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 02:03:08.816263 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0972384-d5d8-41b0-9c98-11cabc4153b3 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:03:08.816721 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0972384-d5d8-41b0-9c98-11cabc4153b3 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:03:08.817124 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0972384-d5d8-41b0-9c98-11cabc4153b3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:03:08.838319 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-d0972384-d5d8-41b0-9c98-11cabc4153b3 None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:03:08.839677 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-d0972384-d5d8-41b0-9c98-11cabc4153b3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:03:08.916963 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1016/2013] 10.222.0.27 () {58 vars in 958 bytes} [Tue Aug 30 02:03:08 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 02:03:08.931122 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94b8f9c2-dc6a-4341-bc86-997f08367d7a None None] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:03:08.931464 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94b8f9c2-dc6a-4341-bc86-997f08367d7a None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:03:08.932251 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94b8f9c2-dc6a-4341-bc86-997f08367d7a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:03:08.955254 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.auth.core [None req-94b8f9c2-dc6a-4341-bc86-997f08367d7a None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81578) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:03:08.956515 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-94b8f9c2-dc6a-4341-bc86-997f08367d7a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:03:08.969561 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 998/2014] 10.222.0.27 () {58 vars in 958 bytes} [Tue Aug 30 02:03:08 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 36 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 02:03:08.971514 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0972384-d5d8-41b0-9c98-11cabc4153b3 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:03:08.983132 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0972384-d5d8-41b0-9c98-11cabc4153b3 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:03:08.988351 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c65ad8e6-ce8d-40b6-924d-11880e6e903b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:03:08.988961 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c65ad8e6-ce8d-40b6-924d-11880e6e903b admin admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:03:08.989240 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c65ad8e6-ce8d-40b6-924d-11880e6e903b admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:03:08.989546 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c65ad8e6-ce8d-40b6-924d-11880e6e903b admin admin] PATH_INFO: `/v3/domains` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:03:08.990207 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c65ad8e6-ce8d-40b6-924d-11880e6e903b admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 02:03:08.990505 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c65ad8e6-ce8d-40b6-924d-11880e6e903b admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:03:08.992308 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c65ad8e6-ce8d-40b6-924d-11880e6e903b admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:03:09.006521 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1017/2015] 10.222.0.27 () {58 vars in 1153 bytes} [Tue Aug 30 02:03:08 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 02:03:09.013800 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94b8f9c2-dc6a-4341-bc86-997f08367d7a None None] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:03:09.023798 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94b8f9c2-dc6a-4341-bc86-997f08367d7a None None] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:03:09.028667 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41ef4d61-8a69-46f9-b384-d71986e86d97 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:03:09.029284 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41ef4d61-8a69-46f9-b384-d71986e86d97 admin admin] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:03:09.030102 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41ef4d61-8a69-46f9-b384-d71986e86d97 admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:03:09.030102 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41ef4d61-8a69-46f9-b384-d71986e86d97 admin admin] PATH_INFO: `/v3/projects` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:03:09.030493 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41ef4d61-8a69-46f9-b384-d71986e86d97 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:03:09.032415 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41ef4d61-8a69-46f9-b384-d71986e86d97 admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:03:09.045686 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=83,keystone:DELETE=15,keystone:INSERT=2 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:03:09.185233 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 999/2016] 10.222.0.27 () {60 vars in 1153 bytes} [Tue Aug 30 02:03:09 2022] POST /identity/v3/projects => generated 407 bytes in 171 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 02:03:09.190139 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c65ad8e6-ce8d-40b6-924d-11880e6e903b admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:03:09.201174 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c65ad8e6-ce8d-40b6-924d-11880e6e903b admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:03:09.265093 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-394947f4-9258-4b68-8e11-41b1b06bca06 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:03:09.265742 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-394947f4-9258-4b68-8e11-41b1b06bca06 admin admin] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:03:09.266016 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-394947f4-9258-4b68-8e11-41b1b06bca06 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:03:09.266284 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-394947f4-9258-4b68-8e11-41b1b06bca06 admin admin] PATH_INFO: `/v3/users` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:03:09.266965 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-394947f4-9258-4b68-8e11-41b1b06bca06 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:03:09.268967 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-394947f4-9258-4b68-8e11-41b1b06bca06 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:03:09.360389 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1018/2017] 10.222.0.27 () {60 vars in 1144 bytes} [Tue Aug 30 02:03:09 2022] POST /identity/v3/users => generated 304 bytes in 168 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 30 02:03:09.361636 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41ef4d61-8a69-46f9-b384-d71986e86d97 admin admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:03:09.372529 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41ef4d61-8a69-46f9-b384-d71986e86d97 admin admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:03:09.383174 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27eceb72-b471-4a47-9d2e-5e301a25639d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:03:09.383903 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27eceb72-b471-4a47-9d2e-5e301a25639d admin admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:03:09.384187 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27eceb72-b471-4a47-9d2e-5e301a25639d admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:03:09.384573 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27eceb72-b471-4a47-9d2e-5e301a25639d admin admin] PATH_INFO: `/v3/roles` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:03:09.385573 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27eceb72-b471-4a47-9d2e-5e301a25639d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:03:09.387254 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27eceb72-b471-4a47-9d2e-5e301a25639d admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:03:09.401514 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1000/2018] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Aug 30 02:03:09 2022] GET /identity/v3/roles => generated 1420 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 02:03:09.403465 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-394947f4-9258-4b68-8e11-41b1b06bca06 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:03:09.414876 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-394947f4-9258-4b68-8e11-41b1b06bca06 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:03:09.420721 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d51bf4ec-9870-4005-8da0-f98fad29cf2d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:03:09.421350 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d51bf4ec-9870-4005-8da0-f98fad29cf2d admin admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:03:09.427501 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d51bf4ec-9870-4005-8da0-f98fad29cf2d admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:03:09.427501 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d51bf4ec-9870-4005-8da0-f98fad29cf2d admin admin] PATH_INFO: `/v3/roles` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:03:09.427501 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d51bf4ec-9870-4005-8da0-f98fad29cf2d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:03:09.427501 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d51bf4ec-9870-4005-8da0-f98fad29cf2d admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:03:09.435462 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1019/2019] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Aug 30 02:03:09 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 02:03:09.439813 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27eceb72-b471-4a47-9d2e-5e301a25639d admin admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:03:09.450692 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27eceb72-b471-4a47-9d2e-5e301a25639d admin admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:03:09.455644 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05e29666-79ff-4f4e-843b-ec49126aa185 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:03:09.456308 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05e29666-79ff-4f4e-843b-ec49126aa185 admin admin] REQUEST_METHOD: `PUT` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:03:09.456588 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05e29666-79ff-4f4e-843b-ec49126aa185 admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:03:09.456859 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05e29666-79ff-4f4e-843b-ec49126aa185 admin admin] PATH_INFO: `/v3/projects/3a95c49cb1884268bf96ce4032ceac18/users/59a2a0fd2b26433d8e99b98d6284d9c1/roles/1fb81afcb9f245f3b989add94598a063` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:03:09.473397 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05e29666-79ff-4f4e-843b-ec49126aa185 admin admin] RBAC: Authorizing `identity:create_grant(project_id=3a95c49cb1884268bf96ce4032ceac18, user_id=59a2a0fd2b26433d8e99b98d6284d9c1, role_id=1fb81afcb9f245f3b989add94598a063)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:03:09.476128 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05e29666-79ff-4f4e-843b-ec49126aa185 admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:03:09.652169 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1001/2020] 10.222.0.27 () {60 vars in 1483 bytes} [Tue Aug 30 02:03:09 2022] PUT /identity/v3/projects/3a95c49cb1884268bf96ce4032ceac18/users/59a2a0fd2b26433d8e99b98d6284d9c1/roles/1fb81afcb9f245f3b989add94598a063 => generated 0 bytes in 213 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 02:03:09.664359 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-977f42f6-0ff9-4394-aeb7-9ac0f63e9151 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:03:09.664764 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-977f42f6-0ff9-4394-aeb7-9ac0f63e9151 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:03:09.665211 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-977f42f6-0ff9-4394-aeb7-9ac0f63e9151 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:03:09.694509 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-977f42f6-0ff9-4394-aeb7-9ac0f63e9151 None None] MFA Rules not processed for user `59a2a0fd2b26433d8e99b98d6284d9c1`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:03:09.696020 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-977f42f6-0ff9-4394-aeb7-9ac0f63e9151 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:03:09.725394 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:03:09.779525 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1020/2021] 10.222.0.27 () {58 vars in 958 bytes} [Tue Aug 30 02:03:09 2022] POST /identity/v3/auth/tokens => generated 3284 bytes in 117 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 02:03:09.784354 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05e29666-79ff-4f4e-843b-ec49126aa185 admin admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:03:09.857892 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05e29666-79ff-4f4e-843b-ec49126aa185 admin admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:03:09.862835 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fe803ad-3a82-4a62-b963-f8fa73bd6380 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:03:09.863461 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fe803ad-3a82-4a62-b963-f8fa73bd6380 admin admin] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:03:09.863769 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fe803ad-3a82-4a62-b963-f8fa73bd6380 admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:03:09.864040 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fe803ad-3a82-4a62-b963-f8fa73bd6380 admin admin] PATH_INFO: `/v3/projects` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:03:09.864691 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fe803ad-3a82-4a62-b963-f8fa73bd6380 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:03:09.866694 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fe803ad-3a82-4a62-b963-f8fa73bd6380 admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:03:10.003537 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1002/2022] 10.222.0.27 () {60 vars in 1153 bytes} [Tue Aug 30 02:03:09 2022] POST /identity/v3/projects => generated 407 bytes in 217 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 02:03:10.006529 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-977f42f6-0ff9-4394-aeb7-9ac0f63e9151 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:03:10.020669 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-977f42f6-0ff9-4394-aeb7-9ac0f63e9151 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:03:10.095384 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df34c2f9-0d05-48a3-808d-0074ad7a2b32 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:03:10.096155 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df34c2f9-0d05-48a3-808d-0074ad7a2b32 admin admin] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:03:10.096681 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df34c2f9-0d05-48a3-808d-0074ad7a2b32 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:03:10.096960 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df34c2f9-0d05-48a3-808d-0074ad7a2b32 admin admin] PATH_INFO: `/v3/users` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:03:10.097643 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df34c2f9-0d05-48a3-808d-0074ad7a2b32 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:03:10.099699 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df34c2f9-0d05-48a3-808d-0074ad7a2b32 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:03:10.271449 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1021/2023] 10.222.0.27 () {60 vars in 1144 bytes} [Tue Aug 30 02:03:10 2022] POST /identity/v3/users => generated 310 bytes in 266 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 02:03:10.276109 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fe803ad-3a82-4a62-b963-f8fa73bd6380 admin admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:03:10.294256 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fe803ad-3a82-4a62-b963-f8fa73bd6380 admin admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:03:10.300218 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce88bbbe-2031-4cca-8909-7b3972f38fb4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:03:10.300916 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce88bbbe-2031-4cca-8909-7b3972f38fb4 admin admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:03:10.301284 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce88bbbe-2031-4cca-8909-7b3972f38fb4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:03:10.301654 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce88bbbe-2031-4cca-8909-7b3972f38fb4 admin admin] PATH_INFO: `/v3/roles` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:03:10.302402 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce88bbbe-2031-4cca-8909-7b3972f38fb4 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:03:10.304047 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce88bbbe-2031-4cca-8909-7b3972f38fb4 admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:03:10.315575 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1003/2024] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Aug 30 02:03:10 2022] GET /identity/v3/roles => generated 1420 bytes in 41 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 02:03:10.319928 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df34c2f9-0d05-48a3-808d-0074ad7a2b32 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:03:10.331325 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df34c2f9-0d05-48a3-808d-0074ad7a2b32 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:03:10.336138 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3c4ab10-ff1b-499b-9f3b-c07098e4728a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:03:10.336805 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3c4ab10-ff1b-499b-9f3b-c07098e4728a admin admin] REQUEST_METHOD: `PUT` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:03:10.337112 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3c4ab10-ff1b-499b-9f3b-c07098e4728a admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:03:10.337389 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3c4ab10-ff1b-499b-9f3b-c07098e4728a admin admin] PATH_INFO: `/v3/projects/b81ffcadbfd8409d9ab2016ea65f1fce/users/ea3c48a3a5de4d0baa891d522dbf9c1f/roles/ead5fac86a0747ea8823a82efcea8ed7` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:03:10.363012 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3c4ab10-ff1b-499b-9f3b-c07098e4728a admin admin] RBAC: Authorizing `identity:create_grant(project_id=b81ffcadbfd8409d9ab2016ea65f1fce, user_id=ea3c48a3a5de4d0baa891d522dbf9c1f, role_id=ead5fac86a0747ea8823a82efcea8ed7)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:03:10.365904 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3c4ab10-ff1b-499b-9f3b-c07098e4728a admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:03:10.510046 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1022/2025] 10.222.0.27 () {60 vars in 1483 bytes} [Tue Aug 30 02:03:10 2022] PUT /identity/v3/projects/b81ffcadbfd8409d9ab2016ea65f1fce/users/ea3c48a3a5de4d0baa891d522dbf9c1f/roles/ead5fac86a0747ea8823a82efcea8ed7 => generated 0 bytes in 190 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 02:03:10.514219 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce88bbbe-2031-4cca-8909-7b3972f38fb4 admin admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:03:10.525068 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce88bbbe-2031-4cca-8909-7b3972f38fb4 admin admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:03:10.595119 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c569ca99-224a-4aac-b47f-5ca76d0a5856 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:03:10.595802 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c569ca99-224a-4aac-b47f-5ca76d0a5856 admin admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:03:10.596102 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c569ca99-224a-4aac-b47f-5ca76d0a5856 admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:03:10.596415 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c569ca99-224a-4aac-b47f-5ca76d0a5856 admin admin] PATH_INFO: `/v3/roles` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:03:10.597070 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c569ca99-224a-4aac-b47f-5ca76d0a5856 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:03:10.598759 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c569ca99-224a-4aac-b47f-5ca76d0a5856 admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:03:10.611700 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1004/2026] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Aug 30 02:03:10 2022] GET /identity/v3/roles => generated 1420 bytes in 98 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 02:03:10.616175 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3c4ab10-ff1b-499b-9f3b-c07098e4728a admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:03:10.628817 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3c4ab10-ff1b-499b-9f3b-c07098e4728a admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:03:10.634432 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-953a153e-5f04-4aba-a560-f2c72b284746 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:03:10.634886 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-953a153e-5f04-4aba-a560-f2c72b284746 admin admin] REQUEST_METHOD: `PUT` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:03:10.635297 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-953a153e-5f04-4aba-a560-f2c72b284746 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:03:10.635439 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-953a153e-5f04-4aba-a560-f2c72b284746 admin admin] PATH_INFO: `/v3/projects/b81ffcadbfd8409d9ab2016ea65f1fce/users/ea3c48a3a5de4d0baa891d522dbf9c1f/roles/ead5fac86a0747ea8823a82efcea8ed7` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:03:10.637631 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-953a153e-5f04-4aba-a560-f2c72b284746 admin admin] RBAC: Authorizing `identity:create_grant(project_id=b81ffcadbfd8409d9ab2016ea65f1fce, user_id=ea3c48a3a5de4d0baa891d522dbf9c1f, role_id=ead5fac86a0747ea8823a82efcea8ed7)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:03:10.640392 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-953a153e-5f04-4aba-a560-f2c72b284746 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:03:10.651328 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1023/2027] 10.222.0.27 () {60 vars in 1483 bytes} [Tue Aug 30 02:03:10 2022] PUT /identity/v3/projects/b81ffcadbfd8409d9ab2016ea65f1fce/users/ea3c48a3a5de4d0baa891d522dbf9c1f/roles/ead5fac86a0747ea8823a82efcea8ed7 => generated 0 bytes in 36 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 02:03:10.661847 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0859dda6-0f83-4b63-9af2-4b6fe62bdcfa None None] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:03:10.662387 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0859dda6-0f83-4b63-9af2-4b6fe62bdcfa None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:03:10.662387 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0859dda6-0f83-4b63-9af2-4b6fe62bdcfa None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:03:10.693631 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.auth.core [None req-0859dda6-0f83-4b63-9af2-4b6fe62bdcfa None None] MFA Rules not processed for user `ea3c48a3a5de4d0baa891d522dbf9c1f`. Rule list: `[]` (Enabled: `True`). {{(pid=81578) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:03:10.694885 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-0859dda6-0f83-4b63-9af2-4b6fe62bdcfa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:03:10.716387 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-953a153e-5f04-4aba-a560-f2c72b284746 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:03:10.718611 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-953a153e-5f04-4aba-a560-f2c72b284746 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:03:10.780025 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1005/2028] 10.222.0.27 () {58 vars in 958 bytes} [Tue Aug 30 02:03:10 2022] POST /identity/v3/auth/tokens => generated 3351 bytes in 119 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 02:03:10.793131 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0859dda6-0f83-4b63-9af2-4b6fe62bdcfa None None] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:03:10.798688 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-0859dda6-0f83-4b63-9af2-4b6fe62bdcfa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:03:10.807390 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-953a153e-5f04-4aba-a560-f2c72b284746 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:03:10.814460 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-691dd0f8-a0fd-4fdf-9d15-15923229a8fc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:03:10.815127 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-691dd0f8-a0fd-4fdf-9d15-15923229a8fc admin admin] REQUEST_METHOD: `DELETE` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:03:10.815410 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-691dd0f8-a0fd-4fdf-9d15-15923229a8fc admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:03:10.815688 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-691dd0f8-a0fd-4fdf-9d15-15923229a8fc admin admin] PATH_INFO: `/v3/users/3d9a18e5feca444287b8f01890121f38` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:03:10.817178 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-691dd0f8-a0fd-4fdf-9d15-15923229a8fc admin admin] RBAC: Authorizing `identity:delete_user(user_id=3d9a18e5feca444287b8f01890121f38)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:03:10.824209 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-691dd0f8-a0fd-4fdf-9d15-15923229a8fc admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:03:10.911089 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0859dda6-0f83-4b63-9af2-4b6fe62bdcfa None None] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:03:10.913293 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=91,keystone:DELETE=8,keystone:INSERT=1 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:03:10.919212 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f938a137-6fc2-487f-bccc-83387865a073 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ab936b3767a8417aa80c8c8f1b84b667', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4562f3951fbe465280073e60a11e44e2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:03:10.919212 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f938a137-6fc2-487f-bccc-83387865a073 service cinder] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:03:10.919212 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f938a137-6fc2-487f-bccc-83387865a073 service cinder] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:03:10.919212 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f938a137-6fc2-487f-bccc-83387865a073 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:03:10.926928 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f938a137-6fc2-487f-bccc-83387865a073 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:03:10.928553 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f938a137-6fc2-487f-bccc-83387865a073 service cinder] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:03:10.940416 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1006/2029] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Aug 30 02:03:10 2022] GET /identity/v3/auth/tokens => generated 3351 bytes in 148 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 02:03:11.161932 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-691dd0f8-a0fd-4fdf-9d15-15923229a8fc admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '3d9a18e5feca444287b8f01890121f38'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:03:11.162891 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-691dd0f8-a0fd-4fdf-9d15-15923229a8fc admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '3d9a18e5feca444287b8f01890121f38'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:03:11.163561 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-691dd0f8-a0fd-4fdf-9d15-15923229a8fc admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '3d9a18e5feca444287b8f01890121f38'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:03:11.177836 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-691dd0f8-a0fd-4fdf-9d15-15923229a8fc admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '3d9a18e5feca444287b8f01890121f38'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:03:11.185886 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-691dd0f8-a0fd-4fdf-9d15-15923229a8fc admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '3d9a18e5feca444287b8f01890121f38'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:03:11.293191 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1024/2030] 10.222.0.27 () {58 vars in 1224 bytes} [Tue Aug 30 02:03:10 2022] DELETE /identity/v3/users/3d9a18e5feca444287b8f01890121f38 => generated 0 bytes in 577 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 02:03:11.299978 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df5ebc36-ec5d-4c21-b25a-1636635869ef None None] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:03:11.300269 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df5ebc36-ec5d-4c21-b25a-1636635869ef None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:03:11.300540 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df5ebc36-ec5d-4c21-b25a-1636635869ef None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:03:11.349279 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.auth.core [None req-df5ebc36-ec5d-4c21-b25a-1636635869ef None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81578) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:03:11.350741 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-df5ebc36-ec5d-4c21-b25a-1636635869ef None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:03:11.413179 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1007/2031] 10.222.0.27 () {58 vars in 958 bytes} [Tue Aug 30 02:03:11 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 114 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 02:03:11.423401 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-691dd0f8-a0fd-4fdf-9d15-15923229a8fc admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:03:11.426717 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-691dd0f8-a0fd-4fdf-9d15-15923229a8fc admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:03:11.487946 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-691dd0f8-a0fd-4fdf-9d15-15923229a8fc admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:03:11.499989 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a802a3d5-d7fc-437e-ab97-21c8297b9b22 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f3ea1580c56f4814a92bbb2acf542905', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4562f3951fbe465280073e60a11e44e2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:03:11.500645 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a802a3d5-d7fc-437e-ab97-21c8297b9b22 service neutron] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:03:11.500930 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a802a3d5-d7fc-437e-ab97-21c8297b9b22 service neutron] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:03:11.501195 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a802a3d5-d7fc-437e-ab97-21c8297b9b22 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:03:11.511891 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a802a3d5-d7fc-437e-ab97-21c8297b9b22 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:03:11.513546 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a802a3d5-d7fc-437e-ab97-21c8297b9b22 service neutron] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:03:11.514342 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df5ebc36-ec5d-4c21-b25a-1636635869ef None None] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:03:11.516532 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-df5ebc36-ec5d-4c21-b25a-1636635869ef None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:03:11.530246 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1025/2032] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Aug 30 02:03:11 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 107 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 02:03:11.587068 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df5ebc36-ec5d-4c21-b25a-1636635869ef None None] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:03:11.590919 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f8ed1f9-5b30-4959-b635-6bf174703e03 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ab936b3767a8417aa80c8c8f1b84b667', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4562f3951fbe465280073e60a11e44e2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:03:11.591849 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f8ed1f9-5b30-4959-b635-6bf174703e03 service cinder] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:03:11.592162 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f8ed1f9-5b30-4959-b635-6bf174703e03 service cinder] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:03:11.592510 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f8ed1f9-5b30-4959-b635-6bf174703e03 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:03:11.594467 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-9f8ed1f9-5b30-4959-b635-6bf174703e03 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:03:11.667325 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f8ed1f9-5b30-4959-b635-6bf174703e03 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:03:11.668717 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f8ed1f9-5b30-4959-b635-6bf174703e03 service cinder] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:03:11.679296 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1008/2033] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Aug 30 02:03:11 2022] GET /identity/v3/auth/tokens => generated 3284 bytes in 166 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 02:03:12.084809 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=90,keystone:DELETE=9,keystone:INSERT=1 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:03:12.093201 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a802a3d5-d7fc-437e-ab97-21c8297b9b22 service neutron] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:03:12.094782 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-a802a3d5-d7fc-437e-ab97-21c8297b9b22 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:03:12.112749 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a802a3d5-d7fc-437e-ab97-21c8297b9b22 service neutron] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:03:12.118420 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df6bfceb-84a2-4780-a4af-436595a547f7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:03:12.119456 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df6bfceb-84a2-4780-a4af-436595a547f7 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:03:12.119730 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df6bfceb-84a2-4780-a4af-436595a547f7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:03:12.120031 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df6bfceb-84a2-4780-a4af-436595a547f7 admin admin] PATH_INFO: `/v3/projects/e0bb2b1a596c4e33ba7ab6d57f218e83` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:03:12.121173 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df6bfceb-84a2-4780-a4af-436595a547f7 admin admin] RBAC: Authorizing `identity:delete_project(project_id=e0bb2b1a596c4e33ba7ab6d57f218e83)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:03:12.123058 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df6bfceb-84a2-4780-a4af-436595a547f7 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:03:12.315297 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-df6bfceb-84a2-4780-a4af-436595a547f7 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'e0bb2b1a596c4e33ba7ab6d57f218e83', 'request_id': 'req-df6bfceb-84a2-4780-a4af-436595a547f7'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:03:12.334553 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-df6bfceb-84a2-4780-a4af-436595a547f7 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'e0bb2b1a596c4e33ba7ab6d57f218e83', 'request_id': 'req-df6bfceb-84a2-4780-a4af-436595a547f7'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:03:12.343732 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-df6bfceb-84a2-4780-a4af-436595a547f7 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'e0bb2b1a596c4e33ba7ab6d57f218e83', 'request_id': 'req-df6bfceb-84a2-4780-a4af-436595a547f7'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:03:12.490201 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-df6bfceb-84a2-4780-a4af-436595a547f7 admin admin] The token cache is being invalidate because project e0bb2b1a596c4e33ba7ab6d57f218e83 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81577) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 30 02:03:12.490496 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-df6bfceb-84a2-4780-a4af-436595a547f7 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:03:12.491159 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-df6bfceb-84a2-4780-a4af-436595a547f7 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:03:12.500428 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1026/2034] 10.222.0.27 () {58 vars in 1233 bytes} [Tue Aug 30 02:03:12 2022] DELETE /identity/v3/projects/e0bb2b1a596c4e33ba7ab6d57f218e83 => generated 0 bytes in 400 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 02:03:12.500539 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f8ed1f9-5b30-4959-b635-6bf174703e03 service cinder] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:03:12.519923 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f8ed1f9-5b30-4959-b635-6bf174703e03 service cinder] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:03:12.583338 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec48da1c-f2b4-45c9-adb8-eaf67f7bc2a8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:03:12.583973 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec48da1c-f2b4-45c9-adb8-eaf67f7bc2a8 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:03:12.584250 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec48da1c-f2b4-45c9-adb8-eaf67f7bc2a8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:03:12.584522 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec48da1c-f2b4-45c9-adb8-eaf67f7bc2a8 admin admin] PATH_INFO: `/v3/users/7a9e18a244f446a2af2eab3110cefe8d` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:03:12.585728 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec48da1c-f2b4-45c9-adb8-eaf67f7bc2a8 admin admin] RBAC: Authorizing `identity:delete_user(user_id=7a9e18a244f446a2af2eab3110cefe8d)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:03:12.588347 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec48da1c-f2b4-45c9-adb8-eaf67f7bc2a8 admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:03:12.824893 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-ec48da1c-f2b4-45c9-adb8-eaf67f7bc2a8 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '7a9e18a244f446a2af2eab3110cefe8d'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:03:12.825603 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-ec48da1c-f2b4-45c9-adb8-eaf67f7bc2a8 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '7a9e18a244f446a2af2eab3110cefe8d'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:03:12.941753 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-ec48da1c-f2b4-45c9-adb8-eaf67f7bc2a8 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '7a9e18a244f446a2af2eab3110cefe8d'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:03:12.942687 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-ec48da1c-f2b4-45c9-adb8-eaf67f7bc2a8 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '7a9e18a244f446a2af2eab3110cefe8d'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:03:12.953170 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-ec48da1c-f2b4-45c9-adb8-eaf67f7bc2a8 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '7a9e18a244f446a2af2eab3110cefe8d'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:03:12.976797 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1009/2035] 10.222.0.27 () {58 vars in 1224 bytes} [Tue Aug 30 02:03:12 2022] DELETE /identity/v3/users/7a9e18a244f446a2af2eab3110cefe8d => generated 0 bytes in 479 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 02:03:13.577191 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:DELETE=19,keystone:SELECT=79,keystone:INSERT=2 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:03:13.673272 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df6bfceb-84a2-4780-a4af-436595a547f7 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:03:13.674898 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-df6bfceb-84a2-4780-a4af-436595a547f7 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:03:13.693846 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df6bfceb-84a2-4780-a4af-436595a547f7 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:03:13.771316 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba885f3c-e3e5-42c2-9453-5c9da9b3643d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:03:13.771952 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba885f3c-e3e5-42c2-9453-5c9da9b3643d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:03:13.772229 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba885f3c-e3e5-42c2-9453-5c9da9b3643d admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:03:13.772498 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba885f3c-e3e5-42c2-9453-5c9da9b3643d admin admin] PATH_INFO: `/v3/projects/98da7ed713a147bfbe5aa58a2e59447b` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:03:13.773632 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba885f3c-e3e5-42c2-9453-5c9da9b3643d admin admin] RBAC: Authorizing `identity:delete_project(project_id=98da7ed713a147bfbe5aa58a2e59447b)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:03:13.776158 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba885f3c-e3e5-42c2-9453-5c9da9b3643d admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:03:13.943257 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-ba885f3c-e3e5-42c2-9453-5c9da9b3643d admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '98da7ed713a147bfbe5aa58a2e59447b', 'request_id': 'req-ba885f3c-e3e5-42c2-9453-5c9da9b3643d'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:03:13.949071 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-ba885f3c-e3e5-42c2-9453-5c9da9b3643d admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '98da7ed713a147bfbe5aa58a2e59447b', 'request_id': 'req-ba885f3c-e3e5-42c2-9453-5c9da9b3643d'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:03:13.955229 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-ba885f3c-e3e5-42c2-9453-5c9da9b3643d admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '98da7ed713a147bfbe5aa58a2e59447b', 'request_id': 'req-ba885f3c-e3e5-42c2-9453-5c9da9b3643d'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:03:14.066955 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-ba885f3c-e3e5-42c2-9453-5c9da9b3643d admin admin] The token cache is being invalidate because project 98da7ed713a147bfbe5aa58a2e59447b was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81577) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 30 02:03:14.067241 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-ba885f3c-e3e5-42c2-9453-5c9da9b3643d admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:03:14.067860 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-ba885f3c-e3e5-42c2-9453-5c9da9b3643d admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:03:14.069345 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1027/2036] 10.222.0.27 () {58 vars in 1233 bytes} [Tue Aug 30 02:03:13 2022] DELETE /identity/v3/projects/98da7ed713a147bfbe5aa58a2e59447b => generated 0 bytes in 397 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 02:03:14.075195 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0909b9c6-cde1-4b1e-aac6-801ebc813587 None None] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:03:14.076076 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0909b9c6-cde1-4b1e-aac6-801ebc813587 None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:03:14.076076 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0909b9c6-cde1-4b1e-aac6-801ebc813587 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:03:14.097490 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.auth.core [None req-0909b9c6-cde1-4b1e-aac6-801ebc813587 None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81578) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:03:14.098637 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-0909b9c6-cde1-4b1e-aac6-801ebc813587 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:03:14.174493 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1010/2037] 10.222.0.27 () {58 vars in 958 bytes} [Tue Aug 30 02:03:14 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 30 02:03:14.184820 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30b05ddf-ab11-4c00-a3f4-b229e2a79412 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:03:14.185108 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30b05ddf-ab11-4c00-a3f4-b229e2a79412 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:03:14.185381 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30b05ddf-ab11-4c00-a3f4-b229e2a79412 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:03:14.207744 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-30b05ddf-ab11-4c00-a3f4-b229e2a79412 None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:03:14.209081 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-30b05ddf-ab11-4c00-a3f4-b229e2a79412 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:03:14.218990 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1028/2038] 10.222.0.27 () {58 vars in 958 bytes} [Tue Aug 30 02:03:14 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 35 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 02:03:14.226187 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0909b9c6-cde1-4b1e-aac6-801ebc813587 None None] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:03:14.234909 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0909b9c6-cde1-4b1e-aac6-801ebc813587 None None] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:03:14.239487 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6564b1b2-dd53-4706-b315-8ea62402c55a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:03:14.240095 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6564b1b2-dd53-4706-b315-8ea62402c55a admin admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:03:14.240372 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6564b1b2-dd53-4706-b315-8ea62402c55a admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:03:14.240639 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6564b1b2-dd53-4706-b315-8ea62402c55a admin admin] PATH_INFO: `/v3/domains` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:03:14.241277 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6564b1b2-dd53-4706-b315-8ea62402c55a admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81578) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 02:03:14.241612 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6564b1b2-dd53-4706-b315-8ea62402c55a admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:03:14.243311 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6564b1b2-dd53-4706-b315-8ea62402c55a admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:03:14.258134 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1011/2039] 10.222.0.27 () {58 vars in 1153 bytes} [Tue Aug 30 02:03:14 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 02:03:14.262854 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30b05ddf-ab11-4c00-a3f4-b229e2a79412 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:03:14.273728 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30b05ddf-ab11-4c00-a3f4-b229e2a79412 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:03:14.280023 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-949e69f3-0482-4633-ac62-622b03612239 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:03:14.280645 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-949e69f3-0482-4633-ac62-622b03612239 admin admin] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:03:14.280922 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-949e69f3-0482-4633-ac62-622b03612239 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:03:14.281191 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-949e69f3-0482-4633-ac62-622b03612239 admin admin] PATH_INFO: `/v3/projects` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:03:14.282033 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-949e69f3-0482-4633-ac62-622b03612239 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:03:14.284031 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-949e69f3-0482-4633-ac62-622b03612239 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:03:14.486260 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1029/2040] 10.222.0.27 () {60 vars in 1153 bytes} [Tue Aug 30 02:03:14 2022] POST /identity/v3/projects => generated 387 bytes in 221 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 02:03:14.487379 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6564b1b2-dd53-4706-b315-8ea62402c55a admin admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:03:14.497936 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6564b1b2-dd53-4706-b315-8ea62402c55a admin admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:03:14.554726 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1bf1440-567f-432f-9974-3b792e543d2e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:03:14.555333 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1bf1440-567f-432f-9974-3b792e543d2e admin admin] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:03:14.555617 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1bf1440-567f-432f-9974-3b792e543d2e admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:03:14.555893 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1bf1440-567f-432f-9974-3b792e543d2e admin admin] PATH_INFO: `/v3/users` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:03:14.556967 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1bf1440-567f-432f-9974-3b792e543d2e admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:03:14.558935 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1bf1440-567f-432f-9974-3b792e543d2e admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:03:14.674280 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1012/2041] 10.222.0.27 () {60 vars in 1144 bytes} [Tue Aug 30 02:03:14 2022] POST /identity/v3/users => generated 294 bytes in 186 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 30 02:03:14.677481 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-949e69f3-0482-4633-ac62-622b03612239 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:03:14.688149 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-949e69f3-0482-4633-ac62-622b03612239 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:03:14.692770 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18624a9c-bd6c-41a5-8306-a8cc06ddfe2d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:03:14.693397 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18624a9c-bd6c-41a5-8306-a8cc06ddfe2d admin admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:03:14.693788 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18624a9c-bd6c-41a5-8306-a8cc06ddfe2d admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:03:14.694061 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18624a9c-bd6c-41a5-8306-a8cc06ddfe2d admin admin] PATH_INFO: `/v3/roles` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:03:14.694702 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18624a9c-bd6c-41a5-8306-a8cc06ddfe2d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:03:14.696385 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18624a9c-bd6c-41a5-8306-a8cc06ddfe2d admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:03:14.707090 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1030/2042] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Aug 30 02:03:14 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 02:03:14.711388 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1bf1440-567f-432f-9974-3b792e543d2e admin admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:03:14.723256 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1bf1440-567f-432f-9974-3b792e543d2e admin admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:03:14.728646 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1112dc08-89c9-4379-b22f-1c45e5d0eb14 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:03:14.729436 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1112dc08-89c9-4379-b22f-1c45e5d0eb14 admin admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:03:14.730429 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1112dc08-89c9-4379-b22f-1c45e5d0eb14 admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:03:14.730726 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1112dc08-89c9-4379-b22f-1c45e5d0eb14 admin admin] PATH_INFO: `/v3/roles` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:03:14.732070 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1112dc08-89c9-4379-b22f-1c45e5d0eb14 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:03:14.733670 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1112dc08-89c9-4379-b22f-1c45e5d0eb14 admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:03:14.744675 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1013/2043] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Aug 30 02:03:14 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 30 02:03:14.749673 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18624a9c-bd6c-41a5-8306-a8cc06ddfe2d admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:03:14.760048 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18624a9c-bd6c-41a5-8306-a8cc06ddfe2d admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:03:14.764928 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cde935a-7ce7-483b-b97e-9744b3162d21 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:03:14.765654 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cde935a-7ce7-483b-b97e-9744b3162d21 admin admin] REQUEST_METHOD: `PUT` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:03:14.765955 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cde935a-7ce7-483b-b97e-9744b3162d21 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:03:14.766243 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cde935a-7ce7-483b-b97e-9744b3162d21 admin admin] PATH_INFO: `/v3/projects/d3eb5d32c7e54dca91e4615ce2f7a027/users/92924b8d54d94d92a6a2529b365c063a/roles/1fb81afcb9f245f3b989add94598a063` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:03:14.782732 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7cde935a-7ce7-483b-b97e-9744b3162d21 admin admin] RBAC: Authorizing `identity:create_grant(project_id=d3eb5d32c7e54dca91e4615ce2f7a027, user_id=92924b8d54d94d92a6a2529b365c063a, role_id=1fb81afcb9f245f3b989add94598a063)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:03:14.785432 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7cde935a-7ce7-483b-b97e-9744b3162d21 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:03:14.924379 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1031/2044] 10.222.0.27 () {60 vars in 1483 bytes} [Tue Aug 30 02:03:14 2022] PUT /identity/v3/projects/d3eb5d32c7e54dca91e4615ce2f7a027/users/92924b8d54d94d92a6a2529b365c063a/roles/1fb81afcb9f245f3b989add94598a063 => generated 0 bytes in 175 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 02:03:14.942094 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-989bac6d-1a96-4ae2-90ed-4da8ea31e0f8 None None] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:03:14.942381 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-989bac6d-1a96-4ae2-90ed-4da8ea31e0f8 None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:03:14.942658 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-989bac6d-1a96-4ae2-90ed-4da8ea31e0f8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:03:14.961805 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:03:14.973496 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.auth.core [None req-989bac6d-1a96-4ae2-90ed-4da8ea31e0f8 None None] MFA Rules not processed for user `92924b8d54d94d92a6a2529b365c063a`. Rule list: `[]` (Enabled: `True`). {{(pid=81578) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:03:14.974769 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-989bac6d-1a96-4ae2-90ed-4da8ea31e0f8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:03:15.043995 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1014/2045] 10.222.0.27 () {58 vars in 958 bytes} [Tue Aug 30 02:03:14 2022] POST /identity/v3/auth/tokens => generated 3264 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 02:03:15.048679 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cde935a-7ce7-483b-b97e-9744b3162d21 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:03:15.110164 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cde935a-7ce7-483b-b97e-9744b3162d21 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:03:15.114863 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cccb0ba4-f445-4382-a986-811d325c647c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:03:15.115484 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cccb0ba4-f445-4382-a986-811d325c647c admin admin] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:03:15.115749 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cccb0ba4-f445-4382-a986-811d325c647c admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:03:15.116021 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cccb0ba4-f445-4382-a986-811d325c647c admin admin] PATH_INFO: `/v3/projects` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:03:15.116645 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cccb0ba4-f445-4382-a986-811d325c647c admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:03:15.118705 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cccb0ba4-f445-4382-a986-811d325c647c admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:03:15.278308 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1032/2046] 10.222.0.27 () {60 vars in 1153 bytes} [Tue Aug 30 02:03:15 2022] POST /identity/v3/projects => generated 387 bytes in 227 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 02:03:15.279413 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-989bac6d-1a96-4ae2-90ed-4da8ea31e0f8 None None] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:03:15.294518 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-989bac6d-1a96-4ae2-90ed-4da8ea31e0f8 None None] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:03:15.356723 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98be9c67-26ac-4961-b253-544a1dab22ab admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:03:15.357358 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98be9c67-26ac-4961-b253-544a1dab22ab admin admin] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:03:15.357670 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98be9c67-26ac-4961-b253-544a1dab22ab admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:03:15.357940 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98be9c67-26ac-4961-b253-544a1dab22ab admin admin] PATH_INFO: `/v3/users` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:03:15.358595 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98be9c67-26ac-4961-b253-544a1dab22ab admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:03:15.360550 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98be9c67-26ac-4961-b253-544a1dab22ab admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:03:15.516322 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1015/2047] 10.222.0.27 () {60 vars in 1144 bytes} [Tue Aug 30 02:03:15 2022] POST /identity/v3/users => generated 300 bytes in 237 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 02:03:15.520651 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cccb0ba4-f445-4382-a986-811d325c647c admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:03:15.531889 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cccb0ba4-f445-4382-a986-811d325c647c admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:03:15.537677 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe91f1bb-a493-4a50-8c2a-d01bcbfc438c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:03:15.538393 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe91f1bb-a493-4a50-8c2a-d01bcbfc438c admin admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:03:15.538713 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe91f1bb-a493-4a50-8c2a-d01bcbfc438c admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:03:15.539052 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe91f1bb-a493-4a50-8c2a-d01bcbfc438c admin admin] PATH_INFO: `/v3/roles` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:03:15.539766 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe91f1bb-a493-4a50-8c2a-d01bcbfc438c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:03:15.549941 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe91f1bb-a493-4a50-8c2a-d01bcbfc438c admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:03:15.563065 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1033/2048] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Aug 30 02:03:15 2022] GET /identity/v3/roles => generated 1420 bytes in 43 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 02:03:15.567564 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98be9c67-26ac-4961-b253-544a1dab22ab admin admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:03:15.579033 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98be9c67-26ac-4961-b253-544a1dab22ab admin admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:03:15.583899 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21dd8a96-384f-4584-b2b6-6b776cb51fe5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:03:15.584566 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21dd8a96-384f-4584-b2b6-6b776cb51fe5 admin admin] REQUEST_METHOD: `PUT` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:03:15.584849 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21dd8a96-384f-4584-b2b6-6b776cb51fe5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:03:15.585121 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21dd8a96-384f-4584-b2b6-6b776cb51fe5 admin admin] PATH_INFO: `/v3/projects/4788db495b0c4bc4ae94b9d6b881eab9/users/1c218a8dca414b52875d9e9c82d02f79/roles/ead5fac86a0747ea8823a82efcea8ed7` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:03:15.602441 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21dd8a96-384f-4584-b2b6-6b776cb51fe5 admin admin] RBAC: Authorizing `identity:create_grant(project_id=4788db495b0c4bc4ae94b9d6b881eab9, user_id=1c218a8dca414b52875d9e9c82d02f79, role_id=ead5fac86a0747ea8823a82efcea8ed7)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:03:15.605679 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21dd8a96-384f-4584-b2b6-6b776cb51fe5 admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:03:15.707941 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1016/2049] 10.222.0.27 () {60 vars in 1483 bytes} [Tue Aug 30 02:03:15 2022] PUT /identity/v3/projects/4788db495b0c4bc4ae94b9d6b881eab9/users/1c218a8dca414b52875d9e9c82d02f79/roles/ead5fac86a0747ea8823a82efcea8ed7 => generated 0 bytes in 140 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 02:03:15.717099 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe91f1bb-a493-4a50-8c2a-d01bcbfc438c admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:03:15.727192 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe91f1bb-a493-4a50-8c2a-d01bcbfc438c admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:03:15.794383 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57153bde-0b08-4fbb-bd7d-6a1bd79abba8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:03:15.797061 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57153bde-0b08-4fbb-bd7d-6a1bd79abba8 admin admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:03:15.797061 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57153bde-0b08-4fbb-bd7d-6a1bd79abba8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:03:15.797061 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57153bde-0b08-4fbb-bd7d-6a1bd79abba8 admin admin] PATH_INFO: `/v3/roles` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:03:15.797061 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57153bde-0b08-4fbb-bd7d-6a1bd79abba8 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:03:15.798057 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57153bde-0b08-4fbb-bd7d-6a1bd79abba8 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:03:15.810985 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1034/2050] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Aug 30 02:03:15 2022] GET /identity/v3/roles => generated 1420 bytes in 92 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 02:03:15.812274 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21dd8a96-384f-4584-b2b6-6b776cb51fe5 admin admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:03:15.824554 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21dd8a96-384f-4584-b2b6-6b776cb51fe5 admin admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:03:15.829341 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae73d612-827e-457e-b0ed-8d977c76b327 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:03:15.830156 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae73d612-827e-457e-b0ed-8d977c76b327 admin admin] REQUEST_METHOD: `PUT` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:03:15.830524 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae73d612-827e-457e-b0ed-8d977c76b327 admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:03:15.830889 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae73d612-827e-457e-b0ed-8d977c76b327 admin admin] PATH_INFO: `/v3/projects/4788db495b0c4bc4ae94b9d6b881eab9/users/1c218a8dca414b52875d9e9c82d02f79/roles/ead5fac86a0747ea8823a82efcea8ed7` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:03:15.833062 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae73d612-827e-457e-b0ed-8d977c76b327 admin admin] RBAC: Authorizing `identity:create_grant(project_id=4788db495b0c4bc4ae94b9d6b881eab9, user_id=1c218a8dca414b52875d9e9c82d02f79, role_id=ead5fac86a0747ea8823a82efcea8ed7)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:03:15.835801 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae73d612-827e-457e-b0ed-8d977c76b327 admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:03:15.847502 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1017/2051] 10.222.0.27 () {60 vars in 1483 bytes} [Tue Aug 30 02:03:15 2022] PUT /identity/v3/projects/4788db495b0c4bc4ae94b9d6b881eab9/users/1c218a8dca414b52875d9e9c82d02f79/roles/ead5fac86a0747ea8823a82efcea8ed7 => generated 0 bytes in 36 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 02:03:15.870271 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd27b103-75f7-4e91-a07e-ea1be8dc92fd None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:03:15.870560 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd27b103-75f7-4e91-a07e-ea1be8dc92fd None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:03:15.870834 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd27b103-75f7-4e91-a07e-ea1be8dc92fd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:03:15.901070 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-bd27b103-75f7-4e91-a07e-ea1be8dc92fd None None] MFA Rules not processed for user `1c218a8dca414b52875d9e9c82d02f79`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:03:15.902433 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-bd27b103-75f7-4e91-a07e-ea1be8dc92fd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:03:15.980040 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1035/2052] 10.222.0.27 () {58 vars in 958 bytes} [Tue Aug 30 02:03:15 2022] POST /identity/v3/auth/tokens => generated 3331 bytes in 111 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 30 02:03:15.996697 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae73d612-827e-457e-b0ed-8d977c76b327 admin admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:03:15.998485 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-ae73d612-827e-457e-b0ed-8d977c76b327 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:03:16.062362 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae73d612-827e-457e-b0ed-8d977c76b327 admin admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:03:16.066222 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b52b827-c180-402b-96d0-c17e9372451b service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ab936b3767a8417aa80c8c8f1b84b667', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4562f3951fbe465280073e60a11e44e2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:03:16.066929 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b52b827-c180-402b-96d0-c17e9372451b service cinder] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:03:16.067211 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b52b827-c180-402b-96d0-c17e9372451b service cinder] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:03:16.067485 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b52b827-c180-402b-96d0-c17e9372451b service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:03:16.078034 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b52b827-c180-402b-96d0-c17e9372451b service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:03:16.078906 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=94,keystone:DELETE=6 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:03:16.079763 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b52b827-c180-402b-96d0-c17e9372451b service cinder] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:03:16.090043 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1018/2053] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Aug 30 02:03:15 2022] GET /identity/v3/auth/tokens => generated 3331 bytes in 99 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 02:03:16.170018 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd27b103-75f7-4e91-a07e-ea1be8dc92fd None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:03:16.200269 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=83,keystone:DELETE=15,keystone:INSERT=2 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:03:16.238539 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd27b103-75f7-4e91-a07e-ea1be8dc92fd None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:03:16.246035 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-060638dd-b9db-430f-a990-feeeda15d2b0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:03:16.246691 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-060638dd-b9db-430f-a990-feeeda15d2b0 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:03:16.246996 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-060638dd-b9db-430f-a990-feeeda15d2b0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:03:16.247268 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-060638dd-b9db-430f-a990-feeeda15d2b0 admin admin] PATH_INFO: `/v3/users/92924b8d54d94d92a6a2529b365c063a` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:03:16.248454 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-060638dd-b9db-430f-a990-feeeda15d2b0 admin admin] RBAC: Authorizing `identity:delete_user(user_id=92924b8d54d94d92a6a2529b365c063a)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:03:16.250520 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-060638dd-b9db-430f-a990-feeeda15d2b0 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:03:16.559840 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-060638dd-b9db-430f-a990-feeeda15d2b0 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '92924b8d54d94d92a6a2529b365c063a'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:03:16.560569 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-060638dd-b9db-430f-a990-feeeda15d2b0 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '92924b8d54d94d92a6a2529b365c063a'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:03:16.561216 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-060638dd-b9db-430f-a990-feeeda15d2b0 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '92924b8d54d94d92a6a2529b365c063a'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:03:16.577487 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-060638dd-b9db-430f-a990-feeeda15d2b0 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '92924b8d54d94d92a6a2529b365c063a'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:03:16.584546 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-060638dd-b9db-430f-a990-feeeda15d2b0 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '92924b8d54d94d92a6a2529b365c063a'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:03:16.744654 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1036/2054] 10.222.0.27 () {58 vars in 1224 bytes} [Tue Aug 30 02:03:16 2022] DELETE /identity/v3/users/92924b8d54d94d92a6a2529b365c063a => generated 0 bytes in 575 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 02:03:16.749849 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dde43d4-d0eb-4aae-9863-ae952fba40dd None None] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:03:16.750788 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dde43d4-d0eb-4aae-9863-ae952fba40dd None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:03:16.751075 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dde43d4-d0eb-4aae-9863-ae952fba40dd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:03:16.776538 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.auth.core [None req-2dde43d4-d0eb-4aae-9863-ae952fba40dd None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81578) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:03:16.777731 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-2dde43d4-d0eb-4aae-9863-ae952fba40dd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:03:16.851249 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1019/2055] 10.222.0.27 () {58 vars in 958 bytes} [Tue Aug 30 02:03:16 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 30 02:03:16.864397 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-060638dd-b9db-430f-a990-feeeda15d2b0 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:03:16.866027 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-060638dd-b9db-430f-a990-feeeda15d2b0 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:03:16.955647 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-060638dd-b9db-430f-a990-feeeda15d2b0 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:03:16.959429 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab737092-f73b-4856-a534-87fcf477ca46 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f3ea1580c56f4814a92bbb2acf542905', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4562f3951fbe465280073e60a11e44e2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:03:16.960049 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab737092-f73b-4856-a534-87fcf477ca46 service neutron] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:03:16.960328 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab737092-f73b-4856-a534-87fcf477ca46 service neutron] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:03:16.960606 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab737092-f73b-4856-a534-87fcf477ca46 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:03:16.971730 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab737092-f73b-4856-a534-87fcf477ca46 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:03:16.977124 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab737092-f73b-4856-a534-87fcf477ca46 service neutron] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:03:16.992092 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1037/2056] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Aug 30 02:03:16 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 128 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 02:03:17.393221 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:INSERT=5,keystone:DELETE=1,keystone:SELECT=94 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:03:17.663475 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2dde43d4-d0eb-4aae-9863-ae952fba40dd None None] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:03:17.664980 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-2dde43d4-d0eb-4aae-9863-ae952fba40dd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:03:17.683282 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2dde43d4-d0eb-4aae-9863-ae952fba40dd None None] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:03:17.691905 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b11cf80c-88ec-4d22-807f-227dd4713eb1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:03:17.691905 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b11cf80c-88ec-4d22-807f-227dd4713eb1 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:03:17.691905 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b11cf80c-88ec-4d22-807f-227dd4713eb1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:03:17.691905 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b11cf80c-88ec-4d22-807f-227dd4713eb1 admin admin] PATH_INFO: `/v3/projects/d3eb5d32c7e54dca91e4615ce2f7a027` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:03:17.692704 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b11cf80c-88ec-4d22-807f-227dd4713eb1 admin admin] RBAC: Authorizing `identity:delete_project(project_id=d3eb5d32c7e54dca91e4615ce2f7a027)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:03:17.694611 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b11cf80c-88ec-4d22-807f-227dd4713eb1 admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:03:17.790086 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-b11cf80c-88ec-4d22-807f-227dd4713eb1 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'd3eb5d32c7e54dca91e4615ce2f7a027', 'request_id': 'req-b11cf80c-88ec-4d22-807f-227dd4713eb1'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:03:17.796735 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-b11cf80c-88ec-4d22-807f-227dd4713eb1 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'd3eb5d32c7e54dca91e4615ce2f7a027', 'request_id': 'req-b11cf80c-88ec-4d22-807f-227dd4713eb1'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:03:17.807286 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-b11cf80c-88ec-4d22-807f-227dd4713eb1 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'd3eb5d32c7e54dca91e4615ce2f7a027', 'request_id': 'req-b11cf80c-88ec-4d22-807f-227dd4713eb1'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:03:18.047346 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-b11cf80c-88ec-4d22-807f-227dd4713eb1 admin admin] The token cache is being invalidate because project d3eb5d32c7e54dca91e4615ce2f7a027 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81578) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 30 02:03:18.047346 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-b11cf80c-88ec-4d22-807f-227dd4713eb1 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:03:18.047346 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-b11cf80c-88ec-4d22-807f-227dd4713eb1 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:03:18.047346 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1020/2057] 10.222.0.27 () {58 vars in 1233 bytes} [Tue Aug 30 02:03:17 2022] DELETE /identity/v3/projects/d3eb5d32c7e54dca91e4615ce2f7a027 => generated 0 bytes in 351 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 02:03:18.066901 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab737092-f73b-4856-a534-87fcf477ca46 service neutron] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:03:18.066901 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab737092-f73b-4856-a534-87fcf477ca46 service neutron] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:03:18.102901 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-477be5da-ad7c-4e60-a788-36123d25ec47 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:03:18.103632 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-477be5da-ad7c-4e60-a788-36123d25ec47 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:03:18.104064 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-477be5da-ad7c-4e60-a788-36123d25ec47 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:03:18.104411 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-477be5da-ad7c-4e60-a788-36123d25ec47 admin admin] PATH_INFO: `/v3/users/1c218a8dca414b52875d9e9c82d02f79` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:03:18.105703 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-477be5da-ad7c-4e60-a788-36123d25ec47 admin admin] RBAC: Authorizing `identity:delete_user(user_id=1c218a8dca414b52875d9e9c82d02f79)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:03:18.107679 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-477be5da-ad7c-4e60-a788-36123d25ec47 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:03:18.811160 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-477be5da-ad7c-4e60-a788-36123d25ec47 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '1c218a8dca414b52875d9e9c82d02f79'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:03:18.811864 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-477be5da-ad7c-4e60-a788-36123d25ec47 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '1c218a8dca414b52875d9e9c82d02f79'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:03:18.812474 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-477be5da-ad7c-4e60-a788-36123d25ec47 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '1c218a8dca414b52875d9e9c82d02f79'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:03:18.815688 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=97,keystone:DELETE=2,keystone:INSERT=1 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:03:18.825945 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-477be5da-ad7c-4e60-a788-36123d25ec47 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '1c218a8dca414b52875d9e9c82d02f79'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:03:18.833924 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-477be5da-ad7c-4e60-a788-36123d25ec47 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '1c218a8dca414b52875d9e9c82d02f79'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:03:18.914710 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1038/2058] 10.222.0.27 () {58 vars in 1224 bytes} [Tue Aug 30 02:03:18 2022] DELETE /identity/v3/users/1c218a8dca414b52875d9e9c82d02f79 => generated 0 bytes in 896 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 02:03:19.498370 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b11cf80c-88ec-4d22-807f-227dd4713eb1 admin admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:03:19.500096 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-b11cf80c-88ec-4d22-807f-227dd4713eb1 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:03:19.530744 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b11cf80c-88ec-4d22-807f-227dd4713eb1 admin admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:03:19.739960 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c927abc9-d9ca-40a5-82ed-2b8805598fd0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:03:19.740632 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c927abc9-d9ca-40a5-82ed-2b8805598fd0 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:03:19.740915 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c927abc9-d9ca-40a5-82ed-2b8805598fd0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:03:19.741201 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c927abc9-d9ca-40a5-82ed-2b8805598fd0 admin admin] PATH_INFO: `/v3/projects/4788db495b0c4bc4ae94b9d6b881eab9` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:03:19.752267 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c927abc9-d9ca-40a5-82ed-2b8805598fd0 admin admin] RBAC: Authorizing `identity:delete_project(project_id=4788db495b0c4bc4ae94b9d6b881eab9)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:03:19.754442 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c927abc9-d9ca-40a5-82ed-2b8805598fd0 admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:03:19.956003 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-c927abc9-d9ca-40a5-82ed-2b8805598fd0 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '4788db495b0c4bc4ae94b9d6b881eab9', 'request_id': 'req-c927abc9-d9ca-40a5-82ed-2b8805598fd0'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:03:19.964414 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-c927abc9-d9ca-40a5-82ed-2b8805598fd0 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '4788db495b0c4bc4ae94b9d6b881eab9', 'request_id': 'req-c927abc9-d9ca-40a5-82ed-2b8805598fd0'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:03:19.971555 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-c927abc9-d9ca-40a5-82ed-2b8805598fd0 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '4788db495b0c4bc4ae94b9d6b881eab9', 'request_id': 'req-c927abc9-d9ca-40a5-82ed-2b8805598fd0'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:03:20.128630 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-c927abc9-d9ca-40a5-82ed-2b8805598fd0 admin admin] The token cache is being invalidate because project 4788db495b0c4bc4ae94b9d6b881eab9 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81578) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 30 02:03:20.128925 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-c927abc9-d9ca-40a5-82ed-2b8805598fd0 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:03:20.129808 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-c927abc9-d9ca-40a5-82ed-2b8805598fd0 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:03:20.133786 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1021/2059] 10.222.0.27 () {58 vars in 1233 bytes} [Tue Aug 30 02:03:19 2022] DELETE /identity/v3/projects/4788db495b0c4bc4ae94b9d6b881eab9 => generated 0 bytes in 634 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 02:03:20.138395 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2389c7a-b1bb-46a4-b505-5d986837e69e None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:03:20.139297 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2389c7a-b1bb-46a4-b505-5d986837e69e None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:03:20.139769 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2389c7a-b1bb-46a4-b505-5d986837e69e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:03:20.186042 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-e2389c7a-b1bb-46a4-b505-5d986837e69e None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:03:20.189127 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-e2389c7a-b1bb-46a4-b505-5d986837e69e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:03:20.244746 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:03:20.256607 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1039/2060] 10.222.0.27 () {58 vars in 958 bytes} [Tue Aug 30 02:03:20 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 120 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 02:03:20.264794 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c75a8a73-4f55-4c2e-9c6c-d32cc0eca37d None None] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:03:20.265101 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c75a8a73-4f55-4c2e-9c6c-d32cc0eca37d None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:03:20.265336 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c75a8a73-4f55-4c2e-9c6c-d32cc0eca37d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:03:20.285634 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.auth.core [None req-c75a8a73-4f55-4c2e-9c6c-d32cc0eca37d None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81578) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:03:20.286838 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-c75a8a73-4f55-4c2e-9c6c-d32cc0eca37d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:03:20.295940 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1022/2061] 10.222.0.27 () {58 vars in 958 bytes} [Tue Aug 30 02:03:20 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 02:03:20.301268 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2389c7a-b1bb-46a4-b505-5d986837e69e None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:03:20.312302 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2389c7a-b1bb-46a4-b505-5d986837e69e None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:03:20.319950 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f7eb404-c319-4eef-9394-cc04e07b4fe5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:03:20.320572 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f7eb404-c319-4eef-9394-cc04e07b4fe5 admin admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:03:20.320840 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f7eb404-c319-4eef-9394-cc04e07b4fe5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:03:20.321110 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f7eb404-c319-4eef-9394-cc04e07b4fe5 admin admin] PATH_INFO: `/v3/domains` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:03:20.323820 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f7eb404-c319-4eef-9394-cc04e07b4fe5 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 02:03:20.324127 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f7eb404-c319-4eef-9394-cc04e07b4fe5 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:03:20.326232 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f7eb404-c319-4eef-9394-cc04e07b4fe5 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:03:20.344158 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1040/2062] 10.222.0.27 () {58 vars in 1153 bytes} [Tue Aug 30 02:03:20 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 44 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 02:03:20.349131 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c75a8a73-4f55-4c2e-9c6c-d32cc0eca37d None None] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:03:20.368104 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c75a8a73-4f55-4c2e-9c6c-d32cc0eca37d None None] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:03:20.401102 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77032de9-4479-46f3-80b4-27c0adbbbcf0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:03:20.401102 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77032de9-4479-46f3-80b4-27c0adbbbcf0 admin admin] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:03:20.401102 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77032de9-4479-46f3-80b4-27c0adbbbcf0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:03:20.401102 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77032de9-4479-46f3-80b4-27c0adbbbcf0 admin admin] PATH_INFO: `/v3/projects` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:03:20.401575 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77032de9-4479-46f3-80b4-27c0adbbbcf0 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:03:20.401575 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77032de9-4479-46f3-80b4-27c0adbbbcf0 admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:03:20.569577 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b091a7c1-7285-4a91-9463-1a7d59d6c063 None None] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:03:20.569869 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b091a7c1-7285-4a91-9463-1a7d59d6c063 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:03:20.570345 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b091a7c1-7285-4a91-9463-1a7d59d6c063 None None] PATH_INFO: `/` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:03:20.572054 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1041/2063] 10.222.0.27 () {54 vars in 921 bytes} [Tue Aug 30 02:03:20 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 02:03:20.576234 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1023/2064] 10.222.0.27 () {60 vars in 1153 bytes} [Tue Aug 30 02:03:20 2022] POST /identity/v3/projects => generated 383 bytes in 225 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 02:03:20.577376 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b091a7c1-7285-4a91-9463-1a7d59d6c063 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:03:20.605565 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bfa80a7-e057-4818-afad-f544f60a8cd7 None None] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:03:20.614117 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bfa80a7-e057-4818-afad-f544f60a8cd7 None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:03:20.614117 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bfa80a7-e057-4818-afad-f544f60a8cd7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:03:20.628832 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.auth.core [None req-8bfa80a7-e057-4818-afad-f544f60a8cd7 None None] MFA Rules not processed for user `379536ff06fd4f719899e5dde732e63d`. Rule list: `[]` (Enabled: `True`). {{(pid=81578) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:03:20.630223 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-8bfa80a7-e057-4818-afad-f544f60a8cd7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:03:20.646788 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b091a7c1-7285-4a91-9463-1a7d59d6c063 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:03:20.652137 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ef31807-3feb-40b7-8128-24822ea3040c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:03:20.652743 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ef31807-3feb-40b7-8128-24822ea3040c admin admin] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:03:20.653023 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ef31807-3feb-40b7-8128-24822ea3040c admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:03:20.653297 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ef31807-3feb-40b7-8128-24822ea3040c admin admin] PATH_INFO: `/v3/users` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:03:20.653986 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ef31807-3feb-40b7-8128-24822ea3040c admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:03:20.656260 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ef31807-3feb-40b7-8128-24822ea3040c admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:03:20.702867 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1024/2065] 10.222.0.27 () {58 vars in 1018 bytes} [Tue Aug 30 02:03:20 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 30 02:03:20.718051 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bfa80a7-e057-4818-afad-f544f60a8cd7 None None] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:03:20.719442 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-8bfa80a7-e057-4818-afad-f544f60a8cd7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:03:20.735484 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bfa80a7-e057-4818-afad-f544f60a8cd7 None None] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:03:20.740607 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34896bfe-366d-4a2b-9303-237f0d5796cf service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '379536ff06fd4f719899e5dde732e63d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4562f3951fbe465280073e60a11e44e2', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:03:20.741252 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34896bfe-366d-4a2b-9303-237f0d5796cf service placement] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:03:20.741566 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34896bfe-366d-4a2b-9303-237f0d5796cf service placement] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:03:20.741840 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34896bfe-366d-4a2b-9303-237f0d5796cf service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:03:20.750648 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34896bfe-366d-4a2b-9303-237f0d5796cf service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:03:20.752207 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34896bfe-366d-4a2b-9303-237f0d5796cf service placement] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:03:20.761987 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1025/2066] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Aug 30 02:03:20 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 45 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 30 02:03:20.810544 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34896bfe-366d-4a2b-9303-237f0d5796cf service placement] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:03:20.811972 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-34896bfe-366d-4a2b-9303-237f0d5796cf service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:03:20.831754 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34896bfe-366d-4a2b-9303-237f0d5796cf service placement] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:03:20.840979 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de661992-3243-4fe2-bfa2-6082bcad935b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:03:20.850996 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de661992-3243-4fe2-bfa2-6082bcad935b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:03:20.850996 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de661992-3243-4fe2-bfa2-6082bcad935b admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:03:20.850996 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de661992-3243-4fe2-bfa2-6082bcad935b admin admin] PATH_INFO: `/v3/users/44f9cfa0ebb84922a9af96ba030ee57a` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:03:20.850996 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de661992-3243-4fe2-bfa2-6082bcad935b admin admin] RBAC: Authorizing `identity:delete_user(user_id=44f9cfa0ebb84922a9af96ba030ee57a)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:03:20.850996 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de661992-3243-4fe2-bfa2-6082bcad935b admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:03:20.851479 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1042/2067] 10.222.0.27 () {60 vars in 1144 bytes} [Tue Aug 30 02:03:20 2022] POST /identity/v3/users => generated 292 bytes in 269 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 30 02:03:20.857292 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ef31807-3feb-40b7-8128-24822ea3040c admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:03:20.871466 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ef31807-3feb-40b7-8128-24822ea3040c admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:03:20.875709 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6177745-c36b-482d-bf03-da468f7a9c49 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:03:20.876333 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6177745-c36b-482d-bf03-da468f7a9c49 admin admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:03:20.876615 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6177745-c36b-482d-bf03-da468f7a9c49 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:03:20.877222 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6177745-c36b-482d-bf03-da468f7a9c49 admin admin] PATH_INFO: `/v3/roles` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:03:20.877672 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6177745-c36b-482d-bf03-da468f7a9c49 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:03:20.879746 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6177745-c36b-482d-bf03-da468f7a9c49 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:03:20.905612 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1043/2068] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Aug 30 02:03:20 2022] GET /identity/v3/roles => generated 1420 bytes in 43 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 02:03:20.907739 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6177745-c36b-482d-bf03-da468f7a9c49 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:03:20.918932 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6177745-c36b-482d-bf03-da468f7a9c49 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:03:20.925303 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6f4e43c-abbd-4eaf-b6d2-530dea64d16a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:03:20.926106 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6f4e43c-abbd-4eaf-b6d2-530dea64d16a admin admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:03:20.926504 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6f4e43c-abbd-4eaf-b6d2-530dea64d16a admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:03:20.926838 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6f4e43c-abbd-4eaf-b6d2-530dea64d16a admin admin] PATH_INFO: `/v3/roles` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:03:20.927878 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6f4e43c-abbd-4eaf-b6d2-530dea64d16a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:03:20.944849 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6f4e43c-abbd-4eaf-b6d2-530dea64d16a admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:03:20.944849 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1044/2069] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Aug 30 02:03:20 2022] GET /identity/v3/roles => generated 1420 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 30 02:03:20.948335 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6f4e43c-abbd-4eaf-b6d2-530dea64d16a admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:03:20.958099 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6f4e43c-abbd-4eaf-b6d2-530dea64d16a admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:03:20.964360 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfdbd896-64b4-4eda-872e-699b660d67a6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:03:20.965017 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfdbd896-64b4-4eda-872e-699b660d67a6 admin admin] REQUEST_METHOD: `PUT` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:03:20.965300 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfdbd896-64b4-4eda-872e-699b660d67a6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:03:20.965625 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfdbd896-64b4-4eda-872e-699b660d67a6 admin admin] PATH_INFO: `/v3/projects/7fd4698b5b434297a25d3793ed5908ad/users/37bf48851ea54e17bb13aa3efaf2ab52/roles/1fb81afcb9f245f3b989add94598a063` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:03:20.980607 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dfdbd896-64b4-4eda-872e-699b660d67a6 admin admin] RBAC: Authorizing `identity:create_grant(project_id=7fd4698b5b434297a25d3793ed5908ad, user_id=37bf48851ea54e17bb13aa3efaf2ab52, role_id=1fb81afcb9f245f3b989add94598a063)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:03:20.983319 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dfdbd896-64b4-4eda-872e-699b660d67a6 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:03:21.137622 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1045/2070] 10.222.0.27 () {60 vars in 1483 bytes} [Tue Aug 30 02:03:20 2022] PUT /identity/v3/projects/7fd4698b5b434297a25d3793ed5908ad/users/37bf48851ea54e17bb13aa3efaf2ab52/roles/1fb81afcb9f245f3b989add94598a063 => generated 0 bytes in 190 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 30 02:03:21.147869 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71c1e594-099a-4465-b867-b5e4d9efb60b None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:03:21.148492 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71c1e594-099a-4465-b867-b5e4d9efb60b None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:03:21.148492 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71c1e594-099a-4465-b867-b5e4d9efb60b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:03:21.176899 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-71c1e594-099a-4465-b867-b5e4d9efb60b None None] MFA Rules not processed for user `37bf48851ea54e17bb13aa3efaf2ab52`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:03:21.178148 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-71c1e594-099a-4465-b867-b5e4d9efb60b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:03:21.203807 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-de661992-3243-4fe2-bfa2-6082bcad935b admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '44f9cfa0ebb84922a9af96ba030ee57a'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:03:21.204594 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-de661992-3243-4fe2-bfa2-6082bcad935b admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '44f9cfa0ebb84922a9af96ba030ee57a'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:03:21.255132 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1046/2071] 10.222.0.27 () {58 vars in 958 bytes} [Tue Aug 30 02:03:21 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 30 02:03:21.262713 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71c1e594-099a-4465-b867-b5e4d9efb60b None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:03:21.330329 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71c1e594-099a-4465-b867-b5e4d9efb60b None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:03:21.335048 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71f88dc5-3e5d-4da2-b293-665bd252addf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:03:21.335667 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71f88dc5-3e5d-4da2-b293-665bd252addf admin admin] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:03:21.335948 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71f88dc5-3e5d-4da2-b293-665bd252addf admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:03:21.336221 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71f88dc5-3e5d-4da2-b293-665bd252addf admin admin] PATH_INFO: `/v3/projects` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:03:21.336867 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71f88dc5-3e5d-4da2-b293-665bd252addf admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:03:21.340934 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71f88dc5-3e5d-4da2-b293-665bd252addf admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:03:21.358564 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-de661992-3243-4fe2-bfa2-6082bcad935b admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '44f9cfa0ebb84922a9af96ba030ee57a'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:03:21.359282 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-de661992-3243-4fe2-bfa2-6082bcad935b admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '44f9cfa0ebb84922a9af96ba030ee57a'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:03:21.367268 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-de661992-3243-4fe2-bfa2-6082bcad935b admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '44f9cfa0ebb84922a9af96ba030ee57a'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:03:21.390719 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1026/2072] 10.222.0.27 () {58 vars in 1224 bytes} [Tue Aug 30 02:03:20 2022] DELETE /identity/v3/users/44f9cfa0ebb84922a9af96ba030ee57a => generated 0 bytes in 581 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 02:03:21.463534 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1047/2073] 10.222.0.27 () {60 vars in 1153 bytes} [Tue Aug 30 02:03:21 2022] POST /identity/v3/projects => generated 383 bytes in 201 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 30 02:03:21.473552 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de661992-3243-4fe2-bfa2-6082bcad935b admin admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:03:21.475089 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-de661992-3243-4fe2-bfa2-6082bcad935b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:03:21.510912 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:03:21.546462 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de661992-3243-4fe2-bfa2-6082bcad935b admin admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:03:21.607960 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6247e88-ff23-40c4-beb1-8d49fdb503a2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:03:21.609047 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6247e88-ff23-40c4-beb1-8d49fdb503a2 admin admin] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:03:21.609047 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6247e88-ff23-40c4-beb1-8d49fdb503a2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:03:21.609223 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6247e88-ff23-40c4-beb1-8d49fdb503a2 admin admin] PATH_INFO: `/v3/users` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:03:21.609855 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6247e88-ff23-40c4-beb1-8d49fdb503a2 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:03:21.612137 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6247e88-ff23-40c4-beb1-8d49fdb503a2 admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:03:21.652180 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71f88dc5-3e5d-4da2-b293-665bd252addf admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:03:21.655021 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-71f88dc5-3e5d-4da2-b293-665bd252addf admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:03:21.667804 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71f88dc5-3e5d-4da2-b293-665bd252addf admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:03:21.672556 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cddb168-ce30-485f-a542-12c297d70caf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:03:21.673191 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cddb168-ce30-485f-a542-12c297d70caf admin admin] REQUEST_METHOD: `DELETE` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:03:21.673566 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cddb168-ce30-485f-a542-12c297d70caf admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:03:21.673769 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cddb168-ce30-485f-a542-12c297d70caf admin admin] PATH_INFO: `/v3/projects/bf405bc0c4b5454aa2b868e9cb5fc7e1` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:03:21.674898 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cddb168-ce30-485f-a542-12c297d70caf admin admin] RBAC: Authorizing `identity:delete_project(project_id=bf405bc0c4b5454aa2b868e9cb5fc7e1)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:03:21.676978 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cddb168-ce30-485f-a542-12c297d70caf admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:03:21.759651 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1027/2074] 10.222.0.27 () {60 vars in 1144 bytes} [Tue Aug 30 02:03:21 2022] POST /identity/v3/users => generated 298 bytes in 287 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 02:03:21.763756 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6247e88-ff23-40c4-beb1-8d49fdb503a2 admin admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:03:21.773695 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6247e88-ff23-40c4-beb1-8d49fdb503a2 admin admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:03:21.779691 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3998770f-9879-4224-a585-671d817be3cb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:03:21.780303 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3998770f-9879-4224-a585-671d817be3cb admin admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:03:21.780585 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3998770f-9879-4224-a585-671d817be3cb admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:03:21.780887 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3998770f-9879-4224-a585-671d817be3cb admin admin] PATH_INFO: `/v3/roles` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:03:21.781570 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3998770f-9879-4224-a585-671d817be3cb admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:03:21.783164 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3998770f-9879-4224-a585-671d817be3cb admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:03:21.795679 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-6cddb168-ce30-485f-a542-12c297d70caf admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'bf405bc0c4b5454aa2b868e9cb5fc7e1', 'request_id': 'req-6cddb168-ce30-485f-a542-12c297d70caf'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:03:21.796187 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1028/2075] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Aug 30 02:03:21 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 30 02:03:21.800391 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3998770f-9879-4224-a585-671d817be3cb admin admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:03:21.805605 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-6cddb168-ce30-485f-a542-12c297d70caf admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'bf405bc0c4b5454aa2b868e9cb5fc7e1', 'request_id': 'req-6cddb168-ce30-485f-a542-12c297d70caf'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:03:21.811081 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3998770f-9879-4224-a585-671d817be3cb admin admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:03:21.813859 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-6cddb168-ce30-485f-a542-12c297d70caf admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'bf405bc0c4b5454aa2b868e9cb5fc7e1', 'request_id': 'req-6cddb168-ce30-485f-a542-12c297d70caf'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:03:21.866532 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1dd33581-3137-4079-8589-64369a584194 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:03:21.867211 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dd33581-3137-4079-8589-64369a584194 admin admin] REQUEST_METHOD: `PUT` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:03:21.867496 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dd33581-3137-4079-8589-64369a584194 admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:03:21.867789 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dd33581-3137-4079-8589-64369a584194 admin admin] PATH_INFO: `/v3/projects/512352c41a8140a7ba9922e32fe3555c/users/836fabfdeb1b4d90bdba37fb5415a38b/roles/ead5fac86a0747ea8823a82efcea8ed7` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:03:21.885350 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1dd33581-3137-4079-8589-64369a584194 admin admin] RBAC: Authorizing `identity:create_grant(project_id=512352c41a8140a7ba9922e32fe3555c, user_id=836fabfdeb1b4d90bdba37fb5415a38b, role_id=ead5fac86a0747ea8823a82efcea8ed7)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:03:21.888087 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1dd33581-3137-4079-8589-64369a584194 admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:03:21.941006 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-6cddb168-ce30-485f-a542-12c297d70caf admin admin] The token cache is being invalidate because project bf405bc0c4b5454aa2b868e9cb5fc7e1 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81577) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 30 02:03:21.941302 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-6cddb168-ce30-485f-a542-12c297d70caf admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:03:21.942008 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-6cddb168-ce30-485f-a542-12c297d70caf admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:03:21.944087 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1048/2076] 10.222.0.27 () {58 vars in 1233 bytes} [Tue Aug 30 02:03:21 2022] DELETE /identity/v3/projects/bf405bc0c4b5454aa2b868e9cb5fc7e1 => generated 0 bytes in 293 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 02:03:21.949278 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-723d1e6c-c1c3-40a9-bd93-3a7e6fef3dde None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:03:21.949706 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-723d1e6c-c1c3-40a9-bd93-3a7e6fef3dde None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:03:21.950026 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-723d1e6c-c1c3-40a9-bd93-3a7e6fef3dde None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:03:21.971177 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1029/2077] 10.222.0.27 () {60 vars in 1483 bytes} [Tue Aug 30 02:03:21 2022] PUT /identity/v3/projects/512352c41a8140a7ba9922e32fe3555c/users/836fabfdeb1b4d90bdba37fb5415a38b/roles/ead5fac86a0747ea8823a82efcea8ed7 => generated 0 bytes in 171 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 30 02:03:21.974375 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-723d1e6c-c1c3-40a9-bd93-3a7e6fef3dde None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:03:21.975127 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1dd33581-3137-4079-8589-64369a584194 admin admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:03:21.975791 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-723d1e6c-c1c3-40a9-bd93-3a7e6fef3dde None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:03:21.976863 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-1dd33581-3137-4079-8589-64369a584194 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:03:21.986102 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=93,keystone:INSERT=7 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:03:22.068757 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1049/2078] 10.222.0.27 () {58 vars in 958 bytes} [Tue Aug 30 02:03:21 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 120 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 30 02:03:22.077566 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27479331-5399-4c91-815c-30dffa7a9aad None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:03:22.077863 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27479331-5399-4c91-815c-30dffa7a9aad None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:03:22.078244 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27479331-5399-4c91-815c-30dffa7a9aad None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:03:22.083153 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1dd33581-3137-4079-8589-64369a584194 admin admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:03:22.092219 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-809a17d1-0f7b-4ba0-9e39-7cafda9ee3af admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:03:22.093191 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-809a17d1-0f7b-4ba0-9e39-7cafda9ee3af admin admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:03:22.093639 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-809a17d1-0f7b-4ba0-9e39-7cafda9ee3af admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:03:22.103576 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-809a17d1-0f7b-4ba0-9e39-7cafda9ee3af admin admin] PATH_INFO: `/v3/roles` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:03:22.103576 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-809a17d1-0f7b-4ba0-9e39-7cafda9ee3af admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:03:22.103576 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-809a17d1-0f7b-4ba0-9e39-7cafda9ee3af admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:03:22.103824 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-27479331-5399-4c91-815c-30dffa7a9aad None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:03:22.103824 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-27479331-5399-4c91-815c-30dffa7a9aad None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:03:22.114060 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1030/2079] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Aug 30 02:03:21 2022] GET /identity/v3/roles => generated 1420 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 30 02:03:22.122223 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1050/2080] 10.222.0.27 () {58 vars in 958 bytes} [Tue Aug 30 02:03:22 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 46 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 30 02:03:22.125510 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-809a17d1-0f7b-4ba0-9e39-7cafda9ee3af admin admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:03:22.126695 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27479331-5399-4c91-815c-30dffa7a9aad None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:03:22.133264 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-809a17d1-0f7b-4ba0-9e39-7cafda9ee3af admin admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:03:22.139421 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27479331-5399-4c91-815c-30dffa7a9aad None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:03:22.139739 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f097123-c9f6-43db-a5cc-f9cb63a47f19 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:03:22.139739 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f097123-c9f6-43db-a5cc-f9cb63a47f19 admin admin] REQUEST_METHOD: `PUT` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:03:22.140128 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f097123-c9f6-43db-a5cc-f9cb63a47f19 admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:03:22.143430 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f097123-c9f6-43db-a5cc-f9cb63a47f19 admin admin] PATH_INFO: `/v3/projects/512352c41a8140a7ba9922e32fe3555c/users/836fabfdeb1b4d90bdba37fb5415a38b/roles/ead5fac86a0747ea8823a82efcea8ed7` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:03:22.143430 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f097123-c9f6-43db-a5cc-f9cb63a47f19 admin admin] RBAC: Authorizing `identity:create_grant(project_id=512352c41a8140a7ba9922e32fe3555c, user_id=836fabfdeb1b4d90bdba37fb5415a38b, role_id=ead5fac86a0747ea8823a82efcea8ed7)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:03:22.145392 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f097123-c9f6-43db-a5cc-f9cb63a47f19 admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:03:22.145564 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0903eb52-74ef-4558-99c1-649ae4106475 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:03:22.146262 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0903eb52-74ef-4558-99c1-649ae4106475 admin admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:03:22.146638 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0903eb52-74ef-4558-99c1-649ae4106475 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:03:22.146975 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0903eb52-74ef-4558-99c1-649ae4106475 admin admin] PATH_INFO: `/v3/domains` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:03:22.147698 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0903eb52-74ef-4558-99c1-649ae4106475 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 02:03:22.148068 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0903eb52-74ef-4558-99c1-649ae4106475 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:03:22.150830 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0903eb52-74ef-4558-99c1-649ae4106475 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:03:22.156993 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1031/2081] 10.222.0.27 () {60 vars in 1483 bytes} [Tue Aug 30 02:03:22 2022] PUT /identity/v3/projects/512352c41a8140a7ba9922e32fe3555c/users/836fabfdeb1b4d90bdba37fb5415a38b/roles/ead5fac86a0747ea8823a82efcea8ed7 => generated 0 bytes in 35 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 02:03:22.171283 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1051/2082] 10.222.0.27 () {58 vars in 1153 bytes} [Tue Aug 30 02:03:22 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 42 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 02:03:22.171463 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4a50c04-62e4-4c03-a2a7-ab7fe92e40b0 None None] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:03:22.171463 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4a50c04-62e4-4c03-a2a7-ab7fe92e40b0 None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:03:22.171463 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4a50c04-62e4-4c03-a2a7-ab7fe92e40b0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:03:22.172748 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0903eb52-74ef-4558-99c1-649ae4106475 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:03:22.183442 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0903eb52-74ef-4558-99c1-649ae4106475 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:03:22.202038 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.auth.core [None req-b4a50c04-62e4-4c03-a2a7-ab7fe92e40b0 None None] MFA Rules not processed for user `836fabfdeb1b4d90bdba37fb5415a38b`. Rule list: `[]` (Enabled: `True`). {{(pid=81578) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:03:22.203384 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-b4a50c04-62e4-4c03-a2a7-ab7fe92e40b0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:03:22.247015 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21fa72d1-d2ae-4aee-8ece-3660da20f66d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:03:22.247793 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21fa72d1-d2ae-4aee-8ece-3660da20f66d admin admin] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:03:22.248355 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21fa72d1-d2ae-4aee-8ece-3660da20f66d admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:03:22.248568 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21fa72d1-d2ae-4aee-8ece-3660da20f66d admin admin] PATH_INFO: `/v3/projects` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:03:22.249283 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21fa72d1-d2ae-4aee-8ece-3660da20f66d admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:03:22.251448 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21fa72d1-d2ae-4aee-8ece-3660da20f66d admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:03:22.279114 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1032/2083] 10.222.0.27 () {58 vars in 958 bytes} [Tue Aug 30 02:03:22 2022] POST /identity/v3/auth/tokens => generated 3327 bytes in 111 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 30 02:03:22.292271 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4a50c04-62e4-4c03-a2a7-ab7fe92e40b0 None None] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:03:22.293766 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-b4a50c04-62e4-4c03-a2a7-ab7fe92e40b0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:03:22.351688 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4a50c04-62e4-4c03-a2a7-ab7fe92e40b0 None None] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:03:22.355540 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c6d3183-32c4-4ea0-9ba1-b211575ee11b service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ab936b3767a8417aa80c8c8f1b84b667', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4562f3951fbe465280073e60a11e44e2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:03:22.356147 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c6d3183-32c4-4ea0-9ba1-b211575ee11b service cinder] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:03:22.356421 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c6d3183-32c4-4ea0-9ba1-b211575ee11b service cinder] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:03:22.356687 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c6d3183-32c4-4ea0-9ba1-b211575ee11b service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:03:22.365849 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c6d3183-32c4-4ea0-9ba1-b211575ee11b service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:03:22.367253 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c6d3183-32c4-4ea0-9ba1-b211575ee11b service cinder] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:03:22.384604 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1033/2084] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Aug 30 02:03:22 2022] GET /identity/v3/auth/tokens => generated 3327 bytes in 93 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 30 02:03:22.448074 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1052/2085] 10.222.0.27 () {60 vars in 1153 bytes} [Tue Aug 30 02:03:22 2022] POST /identity/v3/projects => generated 403 bytes in 276 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 30 02:03:22.452214 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c6d3183-32c4-4ea0-9ba1-b211575ee11b service cinder] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:03:22.463528 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c6d3183-32c4-4ea0-9ba1-b211575ee11b service cinder] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:03:22.519350 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e681896f-5e0c-4db1-a656-240da8b64b8d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:03:22.519961 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e681896f-5e0c-4db1-a656-240da8b64b8d admin admin] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:03:22.520241 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e681896f-5e0c-4db1-a656-240da8b64b8d admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:03:22.520512 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e681896f-5e0c-4db1-a656-240da8b64b8d admin admin] PATH_INFO: `/v3/users` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:03:22.521151 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e681896f-5e0c-4db1-a656-240da8b64b8d admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:03:22.523098 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e681896f-5e0c-4db1-a656-240da8b64b8d admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:03:22.618620 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21fa72d1-d2ae-4aee-8ece-3660da20f66d admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:03:22.637269 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1034/2086] 10.222.0.27 () {60 vars in 1144 bytes} [Tue Aug 30 02:03:22 2022] POST /identity/v3/users => generated 302 bytes in 182 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 02:03:22.638343 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e681896f-5e0c-4db1-a656-240da8b64b8d admin admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:03:22.648036 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e681896f-5e0c-4db1-a656-240da8b64b8d admin admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:03:22.655442 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ba2ee1e-3757-4981-a385-989eb9d4113e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:03:22.656406 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ba2ee1e-3757-4981-a385-989eb9d4113e admin admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:03:22.656893 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ba2ee1e-3757-4981-a385-989eb9d4113e admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:03:22.657349 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ba2ee1e-3757-4981-a385-989eb9d4113e admin admin] PATH_INFO: `/v3/roles` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:03:22.658228 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ba2ee1e-3757-4981-a385-989eb9d4113e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:03:22.660135 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ba2ee1e-3757-4981-a385-989eb9d4113e admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:03:22.670845 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1035/2087] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Aug 30 02:03:22 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 02:03:22.676285 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ba2ee1e-3757-4981-a385-989eb9d4113e admin admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:03:22.684741 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21fa72d1-d2ae-4aee-8ece-3660da20f66d admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:03:22.687153 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ba2ee1e-3757-4981-a385-989eb9d4113e admin admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:03:22.688538 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95b9df37-da2c-424a-a3a9-814b2c4c7cc7 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ab936b3767a8417aa80c8c8f1b84b667', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4562f3951fbe465280073e60a11e44e2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:03:22.689170 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95b9df37-da2c-424a-a3a9-814b2c4c7cc7 service cinder] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:03:22.689449 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95b9df37-da2c-424a-a3a9-814b2c4c7cc7 service cinder] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:03:22.689867 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95b9df37-da2c-424a-a3a9-814b2c4c7cc7 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:03:22.691711 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-95b9df37-da2c-424a-a3a9-814b2c4c7cc7 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:03:22.693363 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8209c11e-002f-4771-8576-94cdaf41a691 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:03:22.694028 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8209c11e-002f-4771-8576-94cdaf41a691 admin admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:03:22.694307 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8209c11e-002f-4771-8576-94cdaf41a691 admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:03:22.694590 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8209c11e-002f-4771-8576-94cdaf41a691 admin admin] PATH_INFO: `/v3/roles` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:03:22.695239 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8209c11e-002f-4771-8576-94cdaf41a691 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:03:22.697509 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8209c11e-002f-4771-8576-94cdaf41a691 admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:03:22.708969 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1036/2088] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Aug 30 02:03:22 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 30 02:03:22.714579 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8209c11e-002f-4771-8576-94cdaf41a691 admin admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:03:22.728776 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8209c11e-002f-4771-8576-94cdaf41a691 admin admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:03:22.733441 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-125ce9f1-d27e-4494-a839-cb725f6baf4e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:03:22.734122 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-125ce9f1-d27e-4494-a839-cb725f6baf4e admin admin] REQUEST_METHOD: `PUT` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:03:22.734397 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-125ce9f1-d27e-4494-a839-cb725f6baf4e admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:03:22.734661 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-125ce9f1-d27e-4494-a839-cb725f6baf4e admin admin] PATH_INFO: `/v3/projects/9f39ba0f80f149ff94f12a3c452f51dc/users/0b03fb9c23c54b5d8c06ae75db48983d/roles/1fb81afcb9f245f3b989add94598a063` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:03:22.749572 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-125ce9f1-d27e-4494-a839-cb725f6baf4e admin admin] RBAC: Authorizing `identity:create_grant(project_id=9f39ba0f80f149ff94f12a3c452f51dc, user_id=0b03fb9c23c54b5d8c06ae75db48983d, role_id=1fb81afcb9f245f3b989add94598a063)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:03:22.753725 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-125ce9f1-d27e-4494-a839-cb725f6baf4e admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:03:22.760719 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95b9df37-da2c-424a-a3a9-814b2c4c7cc7 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:03:22.762384 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95b9df37-da2c-424a-a3a9-814b2c4c7cc7 service cinder] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:03:22.772109 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1053/2089] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Aug 30 02:03:22 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 154 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 02:03:22.873736 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1037/2090] 10.222.0.27 () {60 vars in 1483 bytes} [Tue Aug 30 02:03:22 2022] PUT /identity/v3/projects/9f39ba0f80f149ff94f12a3c452f51dc/users/0b03fb9c23c54b5d8c06ae75db48983d/roles/1fb81afcb9f245f3b989add94598a063 => generated 0 bytes in 160 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 30 02:03:22.884864 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7a7c0de-9f43-494d-9bd5-4bd211fc40c5 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:03:22.885266 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7a7c0de-9f43-494d-9bd5-4bd211fc40c5 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:03:22.885575 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7a7c0de-9f43-494d-9bd5-4bd211fc40c5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:03:22.916524 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-e7a7c0de-9f43-494d-9bd5-4bd211fc40c5 None None] MFA Rules not processed for user `0b03fb9c23c54b5d8c06ae75db48983d`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:03:22.916524 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-e7a7c0de-9f43-494d-9bd5-4bd211fc40c5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:03:22.986487 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1054/2091] 10.222.0.27 () {58 vars in 958 bytes} [Tue Aug 30 02:03:22 2022] POST /identity/v3/auth/tokens => generated 3280 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 02:03:22.991082 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-125ce9f1-d27e-4494-a839-cb725f6baf4e admin admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:03:23.054856 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-125ce9f1-d27e-4494-a839-cb725f6baf4e admin admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:03:23.062833 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0979448-1553-4e73-a53f-949576d89d5c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:03:23.063506 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0979448-1553-4e73-a53f-949576d89d5c admin admin] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:03:23.063786 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0979448-1553-4e73-a53f-949576d89d5c admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:03:23.064067 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0979448-1553-4e73-a53f-949576d89d5c admin admin] PATH_INFO: `/v3/projects` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:03:23.064719 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0979448-1553-4e73-a53f-949576d89d5c admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:03:23.066748 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0979448-1553-4e73-a53f-949576d89d5c admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:03:23.212868 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1038/2092] 10.222.0.27 () {60 vars in 1153 bytes} [Tue Aug 30 02:03:22 2022] POST /identity/v3/projects => generated 405 bytes in 222 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 02:03:23.217392 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7a7c0de-9f43-494d-9bd5-4bd211fc40c5 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:03:23.229854 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7a7c0de-9f43-494d-9bd5-4bd211fc40c5 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:03:23.256449 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0979448-1553-4e73-a53f-949576d89d5c admin admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:03:23.258253 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-b0979448-1553-4e73-a53f-949576d89d5c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:03:23.300785 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d862470-57c9-434a-a94f-10570fb0d3aa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:03:23.304631 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d862470-57c9-434a-a94f-10570fb0d3aa admin admin] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:03:23.304631 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d862470-57c9-434a-a94f-10570fb0d3aa admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:03:23.304631 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d862470-57c9-434a-a94f-10570fb0d3aa admin admin] PATH_INFO: `/v3/users` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:03:23.304631 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d862470-57c9-434a-a94f-10570fb0d3aa admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:03:23.304631 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d862470-57c9-434a-a94f-10570fb0d3aa admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:03:23.399063 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0979448-1553-4e73-a53f-949576d89d5c admin admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:03:23.403874 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b7d1933-5ced-47e9-9651-111a2c947911 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:03:23.404511 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b7d1933-5ced-47e9-9651-111a2c947911 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:03:23.404794 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b7d1933-5ced-47e9-9651-111a2c947911 admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:03:23.405065 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b7d1933-5ced-47e9-9651-111a2c947911 admin admin] PATH_INFO: `/v3/users/59a2a0fd2b26433d8e99b98d6284d9c1` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:03:23.406224 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b7d1933-5ced-47e9-9651-111a2c947911 admin admin] RBAC: Authorizing `identity:delete_user(user_id=59a2a0fd2b26433d8e99b98d6284d9c1)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:03:23.408127 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b7d1933-5ced-47e9-9651-111a2c947911 admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:03:23.495816 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:03:23.502970 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1055/2093] 10.222.0.27 () {60 vars in 1144 bytes} [Tue Aug 30 02:03:23 2022] POST /identity/v3/users => generated 309 bytes in 286 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 30 02:03:23.506072 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d862470-57c9-434a-a94f-10570fb0d3aa admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:03:23.516556 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d862470-57c9-434a-a94f-10570fb0d3aa admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:03:23.521898 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4eb53652-5e9e-4d06-a25d-7ae1e26fdb20 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:03:23.522608 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4eb53652-5e9e-4d06-a25d-7ae1e26fdb20 admin admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:03:23.522889 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4eb53652-5e9e-4d06-a25d-7ae1e26fdb20 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:03:23.523160 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4eb53652-5e9e-4d06-a25d-7ae1e26fdb20 admin admin] PATH_INFO: `/v3/roles` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:03:23.523888 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4eb53652-5e9e-4d06-a25d-7ae1e26fdb20 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:03:23.526137 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4eb53652-5e9e-4d06-a25d-7ae1e26fdb20 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:03:23.541684 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1056/2094] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Aug 30 02:03:23 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 30 02:03:23.541684 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4eb53652-5e9e-4d06-a25d-7ae1e26fdb20 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:03:23.549191 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4eb53652-5e9e-4d06-a25d-7ae1e26fdb20 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:03:23.553864 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1aebde45-6e6b-417f-9b0c-f995dd9165cb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:03:23.554478 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1aebde45-6e6b-417f-9b0c-f995dd9165cb admin admin] REQUEST_METHOD: `DELETE` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:03:23.554758 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1aebde45-6e6b-417f-9b0c-f995dd9165cb admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:03:23.555030 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1aebde45-6e6b-417f-9b0c-f995dd9165cb admin admin] PATH_INFO: `/v3/users/37bf48851ea54e17bb13aa3efaf2ab52` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:03:23.556313 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1aebde45-6e6b-417f-9b0c-f995dd9165cb admin admin] RBAC: Authorizing `identity:delete_user(user_id=37bf48851ea54e17bb13aa3efaf2ab52)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:03:23.558265 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1aebde45-6e6b-417f-9b0c-f995dd9165cb admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:03:23.623372 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-9b7d1933-5ced-47e9-9651-111a2c947911 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '59a2a0fd2b26433d8e99b98d6284d9c1'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:03:23.623989 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-9b7d1933-5ced-47e9-9651-111a2c947911 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '59a2a0fd2b26433d8e99b98d6284d9c1'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:03:23.798890 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-9b7d1933-5ced-47e9-9651-111a2c947911 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '59a2a0fd2b26433d8e99b98d6284d9c1'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:03:23.799350 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-9b7d1933-5ced-47e9-9651-111a2c947911 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '59a2a0fd2b26433d8e99b98d6284d9c1'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:03:23.806863 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-9b7d1933-5ced-47e9-9651-111a2c947911 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '59a2a0fd2b26433d8e99b98d6284d9c1'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:03:23.825492 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1039/2095] 10.222.0.27 () {58 vars in 1224 bytes} [Tue Aug 30 02:03:23 2022] DELETE /identity/v3/users/59a2a0fd2b26433d8e99b98d6284d9c1 => generated 0 bytes in 569 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 02:03:23.826381 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b7d1933-5ced-47e9-9651-111a2c947911 admin admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:03:23.827752 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-9b7d1933-5ced-47e9-9651-111a2c947911 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:03:23.893567 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b7d1933-5ced-47e9-9651-111a2c947911 admin admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:03:23.898273 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac463e39-b69f-4960-9b3b-d7c3b463a427 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:03:23.899072 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac463e39-b69f-4960-9b3b-d7c3b463a427 admin admin] REQUEST_METHOD: `PUT` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:03:23.900081 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac463e39-b69f-4960-9b3b-d7c3b463a427 admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:03:23.900518 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac463e39-b69f-4960-9b3b-d7c3b463a427 admin admin] PATH_INFO: `/v3/projects/90b501c6b7e5479a874b5b6cdfaeadb4/users/d32cefd106b949b381ecf332a3b713b6/roles/ead5fac86a0747ea8823a82efcea8ed7` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:03:23.916673 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac463e39-b69f-4960-9b3b-d7c3b463a427 admin admin] RBAC: Authorizing `identity:create_grant(project_id=90b501c6b7e5479a874b5b6cdfaeadb4, user_id=d32cefd106b949b381ecf332a3b713b6, role_id=ead5fac86a0747ea8823a82efcea8ed7)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:03:23.919474 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac463e39-b69f-4960-9b3b-d7c3b463a427 admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:03:23.925996 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-1aebde45-6e6b-417f-9b0c-f995dd9165cb admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '37bf48851ea54e17bb13aa3efaf2ab52'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:03:23.926680 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-1aebde45-6e6b-417f-9b0c-f995dd9165cb admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '37bf48851ea54e17bb13aa3efaf2ab52'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:03:23.927295 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-1aebde45-6e6b-417f-9b0c-f995dd9165cb admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '37bf48851ea54e17bb13aa3efaf2ab52'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:03:23.940034 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-1aebde45-6e6b-417f-9b0c-f995dd9165cb admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '37bf48851ea54e17bb13aa3efaf2ab52'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:03:23.947408 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-1aebde45-6e6b-417f-9b0c-f995dd9165cb admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '37bf48851ea54e17bb13aa3efaf2ab52'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:03:24.086870 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1040/2096] 10.222.0.27 () {60 vars in 1483 bytes} [Tue Aug 30 02:03:23 2022] PUT /identity/v3/projects/90b501c6b7e5479a874b5b6cdfaeadb4/users/d32cefd106b949b381ecf332a3b713b6/roles/ead5fac86a0747ea8823a82efcea8ed7 => generated 0 bytes in 260 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 02:03:24.088693 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9947d73f-1b85-4d2c-8220-fe5eb36c0f04 None None] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:03:24.088984 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9947d73f-1b85-4d2c-8220-fe5eb36c0f04 None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:03:24.089355 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9947d73f-1b85-4d2c-8220-fe5eb36c0f04 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:03:24.108211 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.auth.core [None req-9947d73f-1b85-4d2c-8220-fe5eb36c0f04 None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81578) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:03:24.109723 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-9947d73f-1b85-4d2c-8220-fe5eb36c0f04 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:03:24.130200 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1057/2097] 10.222.0.27 () {58 vars in 1224 bytes} [Tue Aug 30 02:03:23 2022] DELETE /identity/v3/users/37bf48851ea54e17bb13aa3efaf2ab52 => generated 0 bytes in 592 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 02:03:24.131524 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1aebde45-6e6b-417f-9b0c-f995dd9165cb admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:03:24.134607 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-1aebde45-6e6b-417f-9b0c-f995dd9165cb admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:03:24.189953 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1041/2098] 10.222.0.27 () {58 vars in 958 bytes} [Tue Aug 30 02:03:24 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 02:03:24.194711 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97967756-c613-4e15-b2d7-5753637dc80b None None] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:03:24.195163 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97967756-c613-4e15-b2d7-5753637dc80b None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:03:24.196230 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97967756-c613-4e15-b2d7-5753637dc80b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:03:24.221296 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.auth.core [None req-97967756-c613-4e15-b2d7-5753637dc80b None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81578) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:03:24.221801 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1aebde45-6e6b-417f-9b0c-f995dd9165cb admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:03:24.222516 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-97967756-c613-4e15-b2d7-5753637dc80b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:03:24.226611 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf77c380-b458-4be1-adb5-2873397581ad admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:03:24.227256 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf77c380-b458-4be1-adb5-2873397581ad admin admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:03:24.227722 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf77c380-b458-4be1-adb5-2873397581ad admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:03:24.228088 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf77c380-b458-4be1-adb5-2873397581ad admin admin] PATH_INFO: `/v3/roles` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:03:24.228791 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf77c380-b458-4be1-adb5-2873397581ad admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:03:24.230469 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf77c380-b458-4be1-adb5-2873397581ad admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:03:24.231010 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1042/2099] 10.222.0.27 () {58 vars in 958 bytes} [Tue Aug 30 02:03:24 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 40 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 02:03:24.258946 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97967756-c613-4e15-b2d7-5753637dc80b None None] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:03:24.259229 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:03:24.259360 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-97967756-c613-4e15-b2d7-5753637dc80b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:03:24.259460 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1058/2100] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Aug 30 02:03:24 2022] GET /identity/v3/roles => generated 1420 bytes in 113 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 02:03:24.264911 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf77c380-b458-4be1-adb5-2873397581ad admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:03:24.275119 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf77c380-b458-4be1-adb5-2873397581ad admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:03:24.279843 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2cee941-8e7f-4b1a-8e1c-49899424c591 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:03:24.280499 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2cee941-8e7f-4b1a-8e1c-49899424c591 admin admin] REQUEST_METHOD: `PUT` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:03:24.280777 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2cee941-8e7f-4b1a-8e1c-49899424c591 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:03:24.281052 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2cee941-8e7f-4b1a-8e1c-49899424c591 admin admin] PATH_INFO: `/v3/projects/90b501c6b7e5479a874b5b6cdfaeadb4/users/d32cefd106b949b381ecf332a3b713b6/roles/ead5fac86a0747ea8823a82efcea8ed7` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:03:24.283062 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2cee941-8e7f-4b1a-8e1c-49899424c591 admin admin] RBAC: Authorizing `identity:create_grant(project_id=90b501c6b7e5479a874b5b6cdfaeadb4, user_id=d32cefd106b949b381ecf332a3b713b6, role_id=ead5fac86a0747ea8823a82efcea8ed7)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:03:24.285852 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2cee941-8e7f-4b1a-8e1c-49899424c591 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:03:24.298545 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97967756-c613-4e15-b2d7-5753637dc80b None None] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:03:24.300529 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1059/2101] 10.222.0.27 () {60 vars in 1483 bytes} [Tue Aug 30 02:03:24 2022] PUT /identity/v3/projects/90b501c6b7e5479a874b5b6cdfaeadb4/users/d32cefd106b949b381ecf332a3b713b6/roles/ead5fac86a0747ea8823a82efcea8ed7 => generated 0 bytes in 36 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 30 02:03:24.301402 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2cee941-8e7f-4b1a-8e1c-49899424c591 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:03:24.302684 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-d2cee941-8e7f-4b1a-8e1c-49899424c591 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:03:24.363449 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df2cad02-8f0f-43e2-8c1c-411ec3122b2c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f3ea1580c56f4814a92bbb2acf542905', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4562f3951fbe465280073e60a11e44e2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:03:24.364091 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df2cad02-8f0f-43e2-8c1c-411ec3122b2c service neutron] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:03:24.364430 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df2cad02-8f0f-43e2-8c1c-411ec3122b2c service neutron] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:03:24.364708 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df2cad02-8f0f-43e2-8c1c-411ec3122b2c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:03:24.381037 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2cee941-8e7f-4b1a-8e1c-49899424c591 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:03:24.385371 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e346b03-2a14-4720-be18-1e588f093ad0 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f3ea1580c56f4814a92bbb2acf542905', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4562f3951fbe465280073e60a11e44e2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:03:24.386019 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e346b03-2a14-4720-be18-1e588f093ad0 service neutron] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:03:24.386423 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e346b03-2a14-4720-be18-1e588f093ad0 service neutron] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:03:24.386701 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e346b03-2a14-4720-be18-1e588f093ad0 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:03:24.450357 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df2cad02-8f0f-43e2-8c1c-411ec3122b2c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:03:24.452012 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df2cad02-8f0f-43e2-8c1c-411ec3122b2c service neutron] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:03:24.463543 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1043/2102] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Aug 30 02:03:24 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 232 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 02:03:24.472463 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c94ed8bf-9de7-4a37-99c0-3b080095bbed None None] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:03:24.472898 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c94ed8bf-9de7-4a37-99c0-3b080095bbed None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:03:24.473703 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c94ed8bf-9de7-4a37-99c0-3b080095bbed None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:03:24.476360 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e346b03-2a14-4720-be18-1e588f093ad0 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:03:24.478720 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e346b03-2a14-4720-be18-1e588f093ad0 service neutron] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:03:24.495540 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1060/2103] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Aug 30 02:03:24 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 195 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 02:03:24.508182 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.auth.core [None req-c94ed8bf-9de7-4a37-99c0-3b080095bbed None None] MFA Rules not processed for user `d32cefd106b949b381ecf332a3b713b6`. Rule list: `[]` (Enabled: `True`). {{(pid=81578) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:03:24.510952 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-c94ed8bf-9de7-4a37-99c0-3b080095bbed None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:03:24.616520 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1044/2104] 10.222.0.27 () {58 vars in 958 bytes} [Tue Aug 30 02:03:24 2022] POST /identity/v3/auth/tokens => generated 3349 bytes in 146 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 02:03:24.628865 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e346b03-2a14-4720-be18-1e588f093ad0 service neutron] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:03:24.630383 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-4e346b03-2a14-4720-be18-1e588f093ad0 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:03:24.731302 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e346b03-2a14-4720-be18-1e588f093ad0 service neutron] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:03:24.738955 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7aae520-46b2-4633-af9d-e214e6c41754 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ab936b3767a8417aa80c8c8f1b84b667', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4562f3951fbe465280073e60a11e44e2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:03:24.739753 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7aae520-46b2-4633-af9d-e214e6c41754 service cinder] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:03:24.740101 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7aae520-46b2-4633-af9d-e214e6c41754 service cinder] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:03:24.740365 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7aae520-46b2-4633-af9d-e214e6c41754 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:03:24.763850 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7aae520-46b2-4633-af9d-e214e6c41754 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:03:24.765734 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7aae520-46b2-4633-af9d-e214e6c41754 service cinder] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:03:24.777288 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1061/2105] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Aug 30 02:03:24 2022] GET /identity/v3/auth/tokens => generated 3349 bytes in 149 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 30 02:03:24.980778 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c94ed8bf-9de7-4a37-99c0-3b080095bbed None None] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:03:24.994191 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c94ed8bf-9de7-4a37-99c0-3b080095bbed None None] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:03:25.001376 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82f7f10d-8539-4d3f-b0fa-feb7db0aeee1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:03:25.002115 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82f7f10d-8539-4d3f-b0fa-feb7db0aeee1 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:03:25.002736 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82f7f10d-8539-4d3f-b0fa-feb7db0aeee1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:03:25.002736 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82f7f10d-8539-4d3f-b0fa-feb7db0aeee1 admin admin] PATH_INFO: `/v3/users/0b03fb9c23c54b5d8c06ae75db48983d` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:03:25.003846 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82f7f10d-8539-4d3f-b0fa-feb7db0aeee1 admin admin] RBAC: Authorizing `identity:delete_user(user_id=0b03fb9c23c54b5d8c06ae75db48983d)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:03:25.005958 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82f7f10d-8539-4d3f-b0fa-feb7db0aeee1 admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:03:25.106188 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7aae520-46b2-4633-af9d-e214e6c41754 service cinder] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:03:25.106188 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-d7aae520-46b2-4633-af9d-e214e6c41754 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:03:25.117170 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7aae520-46b2-4633-af9d-e214e6c41754 service cinder] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:03:25.122564 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b50e7f40-591a-449f-a942-78e27dfd95f6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:03:25.123372 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b50e7f40-591a-449f-a942-78e27dfd95f6 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:03:25.123663 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b50e7f40-591a-449f-a942-78e27dfd95f6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:03:25.123937 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b50e7f40-591a-449f-a942-78e27dfd95f6 admin admin] PATH_INFO: `/v3/projects/3a95c49cb1884268bf96ce4032ceac18` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:03:25.125101 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b50e7f40-591a-449f-a942-78e27dfd95f6 admin admin] RBAC: Authorizing `identity:delete_project(project_id=3a95c49cb1884268bf96ce4032ceac18)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:03:25.128023 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b50e7f40-591a-449f-a942-78e27dfd95f6 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:03:25.353342 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:03:25.403264 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-b50e7f40-591a-449f-a942-78e27dfd95f6 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '3a95c49cb1884268bf96ce4032ceac18', 'request_id': 'req-b50e7f40-591a-449f-a942-78e27dfd95f6'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:03:25.404698 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-82f7f10d-8539-4d3f-b0fa-feb7db0aeee1 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '0b03fb9c23c54b5d8c06ae75db48983d'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:03:25.405357 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-82f7f10d-8539-4d3f-b0fa-feb7db0aeee1 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '0b03fb9c23c54b5d8c06ae75db48983d'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:03:25.409586 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-b50e7f40-591a-449f-a942-78e27dfd95f6 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '3a95c49cb1884268bf96ce4032ceac18', 'request_id': 'req-b50e7f40-591a-449f-a942-78e27dfd95f6'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:03:25.416070 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-b50e7f40-591a-449f-a942-78e27dfd95f6 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '3a95c49cb1884268bf96ce4032ceac18', 'request_id': 'req-b50e7f40-591a-449f-a942-78e27dfd95f6'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:03:25.489894 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-82f7f10d-8539-4d3f-b0fa-feb7db0aeee1 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '0b03fb9c23c54b5d8c06ae75db48983d'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:03:25.490504 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-82f7f10d-8539-4d3f-b0fa-feb7db0aeee1 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '0b03fb9c23c54b5d8c06ae75db48983d'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:03:25.501465 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-82f7f10d-8539-4d3f-b0fa-feb7db0aeee1 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '0b03fb9c23c54b5d8c06ae75db48983d'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:03:25.519336 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1045/2106] 10.222.0.27 () {58 vars in 1224 bytes} [Tue Aug 30 02:03:24 2022] DELETE /identity/v3/users/0b03fb9c23c54b5d8c06ae75db48983d => generated 0 bytes in 542 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 02:03:25.520914 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82f7f10d-8539-4d3f-b0fa-feb7db0aeee1 admin admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:03:25.522499 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-82f7f10d-8539-4d3f-b0fa-feb7db0aeee1 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:03:25.585290 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82f7f10d-8539-4d3f-b0fa-feb7db0aeee1 admin admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:03:25.590409 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbda7937-61b6-4c3b-9e9a-7e3f3befbd2e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:03:25.591035 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbda7937-61b6-4c3b-9e9a-7e3f3befbd2e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:03:25.591313 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbda7937-61b6-4c3b-9e9a-7e3f3befbd2e admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:03:25.591839 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbda7937-61b6-4c3b-9e9a-7e3f3befbd2e admin admin] PATH_INFO: `/v3/projects/7fd4698b5b434297a25d3793ed5908ad` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:03:25.593003 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbda7937-61b6-4c3b-9e9a-7e3f3befbd2e admin admin] RBAC: Authorizing `identity:delete_project(project_id=7fd4698b5b434297a25d3793ed5908ad)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:03:25.594923 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbda7937-61b6-4c3b-9e9a-7e3f3befbd2e admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:03:25.620532 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-b50e7f40-591a-449f-a942-78e27dfd95f6 admin admin] The token cache is being invalidate because project 3a95c49cb1884268bf96ce4032ceac18 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81577) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 30 02:03:25.620743 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-b50e7f40-591a-449f-a942-78e27dfd95f6 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:03:25.621379 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-b50e7f40-591a-449f-a942-78e27dfd95f6 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:03:25.623024 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1062/2107] 10.222.0.27 () {58 vars in 1233 bytes} [Tue Aug 30 02:03:25 2022] DELETE /identity/v3/projects/3a95c49cb1884268bf96ce4032ceac18 => generated 0 bytes in 525 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 02:03:25.624278 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a2ea69d-35f5-4de9-a06a-339752be828e None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:03:25.624567 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a2ea69d-35f5-4de9-a06a-339752be828e None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:03:25.624842 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a2ea69d-35f5-4de9-a06a-339752be828e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:03:25.644681 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-4a2ea69d-35f5-4de9-a06a-339752be828e None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:03:25.645926 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-4a2ea69d-35f5-4de9-a06a-339752be828e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:03:25.659382 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1063/2108] 10.222.0.27 () {58 vars in 958 bytes} [Tue Aug 30 02:03:25 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 02:03:25.659382 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a2ea69d-35f5-4de9-a06a-339752be828e None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:03:25.659382 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-4a2ea69d-35f5-4de9-a06a-339752be828e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:03:25.667938 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a2ea69d-35f5-4de9-a06a-339752be828e None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:03:25.672580 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19946c78-f394-4c85-b42d-a383d29ca548 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:03:25.673201 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19946c78-f394-4c85-b42d-a383d29ca548 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:03:25.673509 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19946c78-f394-4c85-b42d-a383d29ca548 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:03:25.673787 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19946c78-f394-4c85-b42d-a383d29ca548 admin admin] PATH_INFO: `/v3/users/ea3c48a3a5de4d0baa891d522dbf9c1f` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:03:25.674891 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19946c78-f394-4c85-b42d-a383d29ca548 admin admin] RBAC: Authorizing `identity:delete_user(user_id=ea3c48a3a5de4d0baa891d522dbf9c1f)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:03:25.676785 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19946c78-f394-4c85-b42d-a383d29ca548 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:03:25.746491 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-fbda7937-61b6-4c3b-9e9a-7e3f3befbd2e admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '7fd4698b5b434297a25d3793ed5908ad', 'request_id': 'req-fbda7937-61b6-4c3b-9e9a-7e3f3befbd2e'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:03:25.752101 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-fbda7937-61b6-4c3b-9e9a-7e3f3befbd2e admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '7fd4698b5b434297a25d3793ed5908ad', 'request_id': 'req-fbda7937-61b6-4c3b-9e9a-7e3f3befbd2e'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:03:25.758269 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-fbda7937-61b6-4c3b-9e9a-7e3f3befbd2e admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '7fd4698b5b434297a25d3793ed5908ad', 'request_id': 'req-fbda7937-61b6-4c3b-9e9a-7e3f3befbd2e'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:03:25.858123 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-fbda7937-61b6-4c3b-9e9a-7e3f3befbd2e admin admin] The token cache is being invalidate because project 7fd4698b5b434297a25d3793ed5908ad was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81578) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 30 02:03:25.859956 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-fbda7937-61b6-4c3b-9e9a-7e3f3befbd2e admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:03:25.859956 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-fbda7937-61b6-4c3b-9e9a-7e3f3befbd2e admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:03:25.861334 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1046/2109] 10.222.0.27 () {58 vars in 1233 bytes} [Tue Aug 30 02:03:25 2022] DELETE /identity/v3/projects/7fd4698b5b434297a25d3793ed5908ad => generated 0 bytes in 341 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 02:03:25.862303 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbda7937-61b6-4c3b-9e9a-7e3f3befbd2e admin admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:03:25.863650 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-fbda7937-61b6-4c3b-9e9a-7e3f3befbd2e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:03:25.923689 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbda7937-61b6-4c3b-9e9a-7e3f3befbd2e admin admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:03:25.927492 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3508d6df-1e3c-408c-8f27-5d559b6b43f7 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f3ea1580c56f4814a92bbb2acf542905', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4562f3951fbe465280073e60a11e44e2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:03:25.928101 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3508d6df-1e3c-408c-8f27-5d559b6b43f7 service neutron] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:03:25.928377 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3508d6df-1e3c-408c-8f27-5d559b6b43f7 service neutron] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:03:25.928645 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3508d6df-1e3c-408c-8f27-5d559b6b43f7 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:03:25.931102 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-3508d6df-1e3c-408c-8f27-5d559b6b43f7 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:03:25.963423 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-19946c78-f394-4c85-b42d-a383d29ca548 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'ea3c48a3a5de4d0baa891d522dbf9c1f'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:03:25.964180 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-19946c78-f394-4c85-b42d-a383d29ca548 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'ea3c48a3a5de4d0baa891d522dbf9c1f'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:03:25.964859 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-19946c78-f394-4c85-b42d-a383d29ca548 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'ea3c48a3a5de4d0baa891d522dbf9c1f'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:03:25.979402 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-19946c78-f394-4c85-b42d-a383d29ca548 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'ea3c48a3a5de4d0baa891d522dbf9c1f'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:03:25.986336 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-19946c78-f394-4c85-b42d-a383d29ca548 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'ea3c48a3a5de4d0baa891d522dbf9c1f'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:03:25.995438 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3508d6df-1e3c-408c-8f27-5d559b6b43f7 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:03:25.996889 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3508d6df-1e3c-408c-8f27-5d559b6b43f7 service neutron] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:03:26.006454 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1047/2110] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Aug 30 02:03:25 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 145 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 02:03:26.007304 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3508d6df-1e3c-408c-8f27-5d559b6b43f7 service neutron] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:03:26.009285 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-3508d6df-1e3c-408c-8f27-5d559b6b43f7 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:03:26.021250 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3508d6df-1e3c-408c-8f27-5d559b6b43f7 service neutron] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:03:26.026170 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fce4c07-6cf1-425c-bb6c-9d8322c8fb16 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:03:26.026806 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fce4c07-6cf1-425c-bb6c-9d8322c8fb16 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:03:26.027089 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fce4c07-6cf1-425c-bb6c-9d8322c8fb16 admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:03:26.027361 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fce4c07-6cf1-425c-bb6c-9d8322c8fb16 admin admin] PATH_INFO: `/v3/users/836fabfdeb1b4d90bdba37fb5415a38b` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:03:26.028641 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fce4c07-6cf1-425c-bb6c-9d8322c8fb16 admin admin] RBAC: Authorizing `identity:delete_user(user_id=836fabfdeb1b4d90bdba37fb5415a38b)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:03:26.030632 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fce4c07-6cf1-425c-bb6c-9d8322c8fb16 admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:03:26.101282 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1064/2111] 10.222.0.27 () {58 vars in 1224 bytes} [Tue Aug 30 02:03:25 2022] DELETE /identity/v3/users/ea3c48a3a5de4d0baa891d522dbf9c1f => generated 0 bytes in 446 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 02:03:26.436669 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-5fce4c07-6cf1-425c-bb6c-9d8322c8fb16 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '836fabfdeb1b4d90bdba37fb5415a38b'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:03:26.437760 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-5fce4c07-6cf1-425c-bb6c-9d8322c8fb16 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '836fabfdeb1b4d90bdba37fb5415a38b'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:03:26.592071 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-5fce4c07-6cf1-425c-bb6c-9d8322c8fb16 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '836fabfdeb1b4d90bdba37fb5415a38b'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:03:26.592669 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-5fce4c07-6cf1-425c-bb6c-9d8322c8fb16 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '836fabfdeb1b4d90bdba37fb5415a38b'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:03:26.599144 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19946c78-f394-4c85-b42d-a383d29ca548 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:03:26.603057 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-19946c78-f394-4c85-b42d-a383d29ca548 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:03:26.603211 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-5fce4c07-6cf1-425c-bb6c-9d8322c8fb16 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '836fabfdeb1b4d90bdba37fb5415a38b'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:03:26.623121 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1048/2112] 10.222.0.27 () {58 vars in 1224 bytes} [Tue Aug 30 02:03:26 2022] DELETE /identity/v3/users/836fabfdeb1b4d90bdba37fb5415a38b => generated 0 bytes in 616 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 02:03:26.654784 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fce4c07-6cf1-425c-bb6c-9d8322c8fb16 admin admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:03:26.656271 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-5fce4c07-6cf1-425c-bb6c-9d8322c8fb16 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:03:26.678357 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19946c78-f394-4c85-b42d-a383d29ca548 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:03:26.710541 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:03:26.745442 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22659ef2-537a-4e17-8814-5a27a36a6719 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:03:26.746283 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22659ef2-537a-4e17-8814-5a27a36a6719 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:03:26.746660 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22659ef2-537a-4e17-8814-5a27a36a6719 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:03:26.746820 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fce4c07-6cf1-425c-bb6c-9d8322c8fb16 admin admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:03:26.747119 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22659ef2-537a-4e17-8814-5a27a36a6719 admin admin] PATH_INFO: `/v3/projects/9f39ba0f80f149ff94f12a3c452f51dc` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:03:26.748330 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22659ef2-537a-4e17-8814-5a27a36a6719 admin admin] RBAC: Authorizing `identity:delete_project(project_id=9f39ba0f80f149ff94f12a3c452f51dc)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:03:26.750374 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22659ef2-537a-4e17-8814-5a27a36a6719 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:03:26.752317 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d748519-7f1d-4c06-acb3-aca739c7509f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:03:26.753006 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d748519-7f1d-4c06-acb3-aca739c7509f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:03:26.753333 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d748519-7f1d-4c06-acb3-aca739c7509f admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:03:26.753639 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d748519-7f1d-4c06-acb3-aca739c7509f admin admin] PATH_INFO: `/v3/projects/b81ffcadbfd8409d9ab2016ea65f1fce` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:03:26.754761 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d748519-7f1d-4c06-acb3-aca739c7509f admin admin] RBAC: Authorizing `identity:delete_project(project_id=b81ffcadbfd8409d9ab2016ea65f1fce)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:03:26.758152 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d748519-7f1d-4c06-acb3-aca739c7509f admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:03:26.921707 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-6d748519-7f1d-4c06-acb3-aca739c7509f admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'b81ffcadbfd8409d9ab2016ea65f1fce', 'request_id': 'req-6d748519-7f1d-4c06-acb3-aca739c7509f'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:03:26.928282 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-6d748519-7f1d-4c06-acb3-aca739c7509f admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'b81ffcadbfd8409d9ab2016ea65f1fce', 'request_id': 'req-6d748519-7f1d-4c06-acb3-aca739c7509f'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:03:26.935279 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-6d748519-7f1d-4c06-acb3-aca739c7509f admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'b81ffcadbfd8409d9ab2016ea65f1fce', 'request_id': 'req-6d748519-7f1d-4c06-acb3-aca739c7509f'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:03:26.942198 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-22659ef2-537a-4e17-8814-5a27a36a6719 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '9f39ba0f80f149ff94f12a3c452f51dc', 'request_id': 'req-22659ef2-537a-4e17-8814-5a27a36a6719'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:03:26.952696 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:INSERT=7,keystone:SELECT=93 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:03:26.967041 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-22659ef2-537a-4e17-8814-5a27a36a6719 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '9f39ba0f80f149ff94f12a3c452f51dc', 'request_id': 'req-22659ef2-537a-4e17-8814-5a27a36a6719'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:03:26.980261 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-22659ef2-537a-4e17-8814-5a27a36a6719 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '9f39ba0f80f149ff94f12a3c452f51dc', 'request_id': 'req-22659ef2-537a-4e17-8814-5a27a36a6719'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:03:27.051353 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-6d748519-7f1d-4c06-acb3-aca739c7509f admin admin] The token cache is being invalidate because project b81ffcadbfd8409d9ab2016ea65f1fce was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81578) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 30 02:03:27.051353 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-6d748519-7f1d-4c06-acb3-aca739c7509f admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:03:27.051353 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-6d748519-7f1d-4c06-acb3-aca739c7509f admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:03:27.051353 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1049/2113] 10.222.0.27 () {58 vars in 1233 bytes} [Tue Aug 30 02:03:26 2022] DELETE /identity/v3/projects/b81ffcadbfd8409d9ab2016ea65f1fce => generated 0 bytes in 391 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 02:03:27.055447 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f132392-29e2-41fb-b299-e0a52e833876 None None] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:03:27.055814 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f132392-29e2-41fb-b299-e0a52e833876 None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:03:27.056092 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f132392-29e2-41fb-b299-e0a52e833876 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:03:27.080892 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.auth.core [None req-9f132392-29e2-41fb-b299-e0a52e833876 None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81578) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:03:27.082131 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-9f132392-29e2-41fb-b299-e0a52e833876 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:03:27.169366 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-22659ef2-537a-4e17-8814-5a27a36a6719 admin admin] The token cache is being invalidate because project 9f39ba0f80f149ff94f12a3c452f51dc was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81577) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 30 02:03:27.169719 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-22659ef2-537a-4e17-8814-5a27a36a6719 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:03:27.171365 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-22659ef2-537a-4e17-8814-5a27a36a6719 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:03:27.171841 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1065/2114] 10.222.0.27 () {58 vars in 1233 bytes} [Tue Aug 30 02:03:26 2022] DELETE /identity/v3/projects/9f39ba0f80f149ff94f12a3c452f51dc => generated 0 bytes in 573 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 02:03:27.175656 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22659ef2-537a-4e17-8814-5a27a36a6719 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:03:27.176915 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-22659ef2-537a-4e17-8814-5a27a36a6719 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:03:27.276104 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1050/2115] 10.222.0.27 () {58 vars in 958 bytes} [Tue Aug 30 02:03:27 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 222 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 30 02:03:27.282312 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22659ef2-537a-4e17-8814-5a27a36a6719 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:03:27.288118 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80302755-865f-4ae2-9961-9733efd069dc None None] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:03:27.288595 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80302755-865f-4ae2-9961-9733efd069dc None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:03:27.288741 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29f20468-2cbf-4fe3-9955-83dc5f0a7819 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:03:27.289062 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80302755-865f-4ae2-9961-9733efd069dc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:03:27.289377 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29f20468-2cbf-4fe3-9955-83dc5f0a7819 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:03:27.289799 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29f20468-2cbf-4fe3-9955-83dc5f0a7819 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:03:27.290084 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29f20468-2cbf-4fe3-9955-83dc5f0a7819 admin admin] PATH_INFO: `/v3/users/d32cefd106b949b381ecf332a3b713b6` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:03:27.291216 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29f20468-2cbf-4fe3-9955-83dc5f0a7819 admin admin] RBAC: Authorizing `identity:delete_user(user_id=d32cefd106b949b381ecf332a3b713b6)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:03:27.293151 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29f20468-2cbf-4fe3-9955-83dc5f0a7819 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:03:27.310893 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.auth.core [None req-80302755-865f-4ae2-9961-9733efd069dc None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81578) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:03:27.312098 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-80302755-865f-4ae2-9961-9733efd069dc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:03:27.320927 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1051/2116] 10.222.0.27 () {58 vars in 958 bytes} [Tue Aug 30 02:03:27 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 02:03:27.325809 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80302755-865f-4ae2-9961-9733efd069dc None None] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:03:27.334734 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80302755-865f-4ae2-9961-9733efd069dc None None] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:03:27.339644 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56124c3f-c610-4eb1-977d-6a34194cf299 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:03:27.340215 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56124c3f-c610-4eb1-977d-6a34194cf299 admin admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:03:27.340493 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56124c3f-c610-4eb1-977d-6a34194cf299 admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:03:27.340762 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56124c3f-c610-4eb1-977d-6a34194cf299 admin admin] PATH_INFO: `/v3/domains` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:03:27.341915 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56124c3f-c610-4eb1-977d-6a34194cf299 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81578) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 02:03:27.342492 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56124c3f-c610-4eb1-977d-6a34194cf299 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:03:27.343583 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56124c3f-c610-4eb1-977d-6a34194cf299 admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:03:27.358182 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1052/2117] 10.222.0.27 () {58 vars in 1153 bytes} [Tue Aug 30 02:03:27 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 30 02:03:27.359284 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56124c3f-c610-4eb1-977d-6a34194cf299 admin admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:03:27.363003 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-56124c3f-c610-4eb1-977d-6a34194cf299 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:03:27.376805 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56124c3f-c610-4eb1-977d-6a34194cf299 admin admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:03:27.382275 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8783dd18-dcf7-405d-93fc-d6023166470d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:03:27.382897 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8783dd18-dcf7-405d-93fc-d6023166470d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:03:27.383177 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8783dd18-dcf7-405d-93fc-d6023166470d admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:03:27.383511 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8783dd18-dcf7-405d-93fc-d6023166470d admin admin] PATH_INFO: `/v3/projects/512352c41a8140a7ba9922e32fe3555c` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:03:27.385325 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8783dd18-dcf7-405d-93fc-d6023166470d admin admin] RBAC: Authorizing `identity:delete_project(project_id=512352c41a8140a7ba9922e32fe3555c)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:03:27.387490 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8783dd18-dcf7-405d-93fc-d6023166470d admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:03:27.629578 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-8783dd18-dcf7-405d-93fc-d6023166470d admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '512352c41a8140a7ba9922e32fe3555c', 'request_id': 'req-8783dd18-dcf7-405d-93fc-d6023166470d'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:03:27.630235 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-29f20468-2cbf-4fe3-9955-83dc5f0a7819 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'd32cefd106b949b381ecf332a3b713b6'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:03:27.630879 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-29f20468-2cbf-4fe3-9955-83dc5f0a7819 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'd32cefd106b949b381ecf332a3b713b6'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:03:27.631504 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-29f20468-2cbf-4fe3-9955-83dc5f0a7819 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'd32cefd106b949b381ecf332a3b713b6'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:03:27.635856 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-8783dd18-dcf7-405d-93fc-d6023166470d admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '512352c41a8140a7ba9922e32fe3555c', 'request_id': 'req-8783dd18-dcf7-405d-93fc-d6023166470d'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:03:27.642435 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-8783dd18-dcf7-405d-93fc-d6023166470d admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '512352c41a8140a7ba9922e32fe3555c', 'request_id': 'req-8783dd18-dcf7-405d-93fc-d6023166470d'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:03:27.645041 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-29f20468-2cbf-4fe3-9955-83dc5f0a7819 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'd32cefd106b949b381ecf332a3b713b6'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:03:27.651863 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-29f20468-2cbf-4fe3-9955-83dc5f0a7819 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'd32cefd106b949b381ecf332a3b713b6'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:03:27.985409 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-8783dd18-dcf7-405d-93fc-d6023166470d admin admin] The token cache is being invalidate because project 512352c41a8140a7ba9922e32fe3555c was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81578) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 30 02:03:27.985737 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-8783dd18-dcf7-405d-93fc-d6023166470d admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:03:27.986356 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-8783dd18-dcf7-405d-93fc-d6023166470d admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:03:27.987838 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1053/2118] 10.222.0.27 () {58 vars in 1233 bytes} [Tue Aug 30 02:03:27 2022] DELETE /identity/v3/projects/512352c41a8140a7ba9922e32fe3555c => generated 0 bytes in 629 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 02:03:27.988597 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8783dd18-dcf7-405d-93fc-d6023166470d admin admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:03:27.989904 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-8783dd18-dcf7-405d-93fc-d6023166470d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:03:28.054173 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8783dd18-dcf7-405d-93fc-d6023166470d admin admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:03:28.062256 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38c84eec-5984-416a-b4ac-b8c7221a62a0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:03:28.062878 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38c84eec-5984-416a-b4ac-b8c7221a62a0 admin admin] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:03:28.063387 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38c84eec-5984-416a-b4ac-b8c7221a62a0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:03:28.063872 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38c84eec-5984-416a-b4ac-b8c7221a62a0 admin admin] PATH_INFO: `/v3/projects` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:03:28.064724 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38c84eec-5984-416a-b4ac-b8c7221a62a0 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:03:28.066696 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38c84eec-5984-416a-b4ac-b8c7221a62a0 admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:03:28.119271 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1066/2119] 10.222.0.27 () {58 vars in 1224 bytes} [Tue Aug 30 02:03:27 2022] DELETE /identity/v3/users/d32cefd106b949b381ecf332a3b713b6 => generated 0 bytes in 944 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 30 02:03:28.120605 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93f8b295-753c-463d-9efd-28594aa2ab0a None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:03:28.120889 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93f8b295-753c-463d-9efd-28594aa2ab0a None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:03:28.130710 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93f8b295-753c-463d-9efd-28594aa2ab0a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:03:28.140503 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-93f8b295-753c-463d-9efd-28594aa2ab0a None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:03:28.141727 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-93f8b295-753c-463d-9efd-28594aa2ab0a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:03:28.218715 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1067/2120] 10.222.0.27 () {58 vars in 958 bytes} [Tue Aug 30 02:03:28 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 02:03:28.220959 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-628900ad-1bbe-40ac-b3ce-a708c80def69 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:03:28.221255 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-628900ad-1bbe-40ac-b3ce-a708c80def69 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:03:28.221587 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-628900ad-1bbe-40ac-b3ce-a708c80def69 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:03:28.247785 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-628900ad-1bbe-40ac-b3ce-a708c80def69 None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:03:28.249547 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-628900ad-1bbe-40ac-b3ce-a708c80def69 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:03:28.257252 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1068/2121] 10.222.0.27 () {58 vars in 958 bytes} [Tue Aug 30 02:03:28 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 38 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 30 02:03:28.262053 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-628900ad-1bbe-40ac-b3ce-a708c80def69 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:03:28.271679 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-628900ad-1bbe-40ac-b3ce-a708c80def69 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:03:28.277082 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1054/2122] 10.222.0.27 () {60 vars in 1153 bytes} [Tue Aug 30 02:03:27 2022] POST /identity/v3/projects => generated 377 bytes in 286 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 02:03:28.277384 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe8b1d30-aaf7-4e60-b602-738853f64005 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:03:28.278042 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe8b1d30-aaf7-4e60-b602-738853f64005 admin admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:03:28.278324 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe8b1d30-aaf7-4e60-b602-738853f64005 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:03:28.278600 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe8b1d30-aaf7-4e60-b602-738853f64005 admin admin] PATH_INFO: `/v3/domains` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:03:28.278886 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38c84eec-5984-416a-b4ac-b8c7221a62a0 admin admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:03:28.279270 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe8b1d30-aaf7-4e60-b602-738853f64005 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 02:03:28.279574 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe8b1d30-aaf7-4e60-b602-738853f64005 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:03:28.281197 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe8b1d30-aaf7-4e60-b602-738853f64005 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:03:28.294816 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1069/2123] 10.222.0.27 () {58 vars in 1153 bytes} [Tue Aug 30 02:03:28 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 30 02:03:28.299131 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe8b1d30-aaf7-4e60-b602-738853f64005 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:03:28.309926 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe8b1d30-aaf7-4e60-b602-738853f64005 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:03:28.345731 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38c84eec-5984-416a-b4ac-b8c7221a62a0 admin admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:03:28.350427 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0973bbcd-6f39-4bcb-8b9a-c8560eb6c94b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:03:28.351039 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0973bbcd-6f39-4bcb-8b9a-c8560eb6c94b admin admin] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:03:28.351323 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0973bbcd-6f39-4bcb-8b9a-c8560eb6c94b admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:03:28.351598 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0973bbcd-6f39-4bcb-8b9a-c8560eb6c94b admin admin] PATH_INFO: `/v3/users` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:03:28.352936 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0973bbcd-6f39-4bcb-8b9a-c8560eb6c94b admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:03:28.355257 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0973bbcd-6f39-4bcb-8b9a-c8560eb6c94b admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:03:28.375854 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e837b487-e5ce-4081-ae15-ce5598e4b560 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:03:28.376455 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e837b487-e5ce-4081-ae15-ce5598e4b560 admin admin] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:03:28.376731 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e837b487-e5ce-4081-ae15-ce5598e4b560 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:03:28.376999 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e837b487-e5ce-4081-ae15-ce5598e4b560 admin admin] PATH_INFO: `/v3/projects` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:03:28.377649 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e837b487-e5ce-4081-ae15-ce5598e4b560 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:03:28.379527 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e837b487-e5ce-4081-ae15-ce5598e4b560 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:03:28.506580 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1055/2124] 10.222.0.27 () {60 vars in 1144 bytes} [Tue Aug 30 02:03:28 2022] POST /identity/v3/users => generated 289 bytes in 228 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 02:03:28.510786 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0973bbcd-6f39-4bcb-8b9a-c8560eb6c94b admin admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:03:28.520656 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0973bbcd-6f39-4bcb-8b9a-c8560eb6c94b admin admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:03:28.525934 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-594945c1-019f-4591-9a4f-7fe857f373f9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:03:28.526537 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-594945c1-019f-4591-9a4f-7fe857f373f9 admin admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:03:28.526820 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-594945c1-019f-4591-9a4f-7fe857f373f9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:03:28.527094 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-594945c1-019f-4591-9a4f-7fe857f373f9 admin admin] PATH_INFO: `/v3/roles` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:03:28.527742 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-594945c1-019f-4591-9a4f-7fe857f373f9 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:03:28.529347 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-594945c1-019f-4591-9a4f-7fe857f373f9 admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:03:28.541502 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1056/2125] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Aug 30 02:03:28 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 30 02:03:28.546386 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-594945c1-019f-4591-9a4f-7fe857f373f9 admin admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:03:28.556125 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-594945c1-019f-4591-9a4f-7fe857f373f9 admin admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:03:28.560663 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ef21fff-fee1-4d13-be6b-9aab626ce809 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:03:28.561486 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ef21fff-fee1-4d13-be6b-9aab626ce809 admin admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:03:28.561623 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ef21fff-fee1-4d13-be6b-9aab626ce809 admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:03:28.561812 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ef21fff-fee1-4d13-be6b-9aab626ce809 admin admin] PATH_INFO: `/v3/roles` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:03:28.562447 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ef21fff-fee1-4d13-be6b-9aab626ce809 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:03:28.564014 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ef21fff-fee1-4d13-be6b-9aab626ce809 admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:03:28.575337 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1057/2126] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Aug 30 02:03:28 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 30 02:03:28.581011 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ef21fff-fee1-4d13-be6b-9aab626ce809 admin admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:03:28.583198 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1070/2127] 10.222.0.27 () {60 vars in 1153 bytes} [Tue Aug 30 02:03:28 2022] POST /identity/v3/projects => generated 371 bytes in 284 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 30 02:03:28.591704 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ef21fff-fee1-4d13-be6b-9aab626ce809 admin admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:03:28.602261 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e837b487-e5ce-4081-ae15-ce5598e4b560 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:03:28.651309 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-367f41d8-9ed8-4a39-8111-46245f166868 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:03:28.652367 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-367f41d8-9ed8-4a39-8111-46245f166868 admin admin] REQUEST_METHOD: `PUT` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:03:28.652367 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-367f41d8-9ed8-4a39-8111-46245f166868 admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:03:28.652765 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-367f41d8-9ed8-4a39-8111-46245f166868 admin admin] PATH_INFO: `/v3/projects/14567ca5cc3249f4aef186f05fad1cda/users/b3b5830806e643a1a7577a11a66c554d/roles/1fb81afcb9f245f3b989add94598a063` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:03:28.664440 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e837b487-e5ce-4081-ae15-ce5598e4b560 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:03:28.667823 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-367f41d8-9ed8-4a39-8111-46245f166868 admin admin] RBAC: Authorizing `identity:create_grant(project_id=14567ca5cc3249f4aef186f05fad1cda, user_id=b3b5830806e643a1a7577a11a66c554d, role_id=1fb81afcb9f245f3b989add94598a063)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:03:28.669878 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4afc674e-903e-46c7-a84e-159b181a66b0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:03:28.670481 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4afc674e-903e-46c7-a84e-159b181a66b0 admin admin] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:03:28.670579 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-367f41d8-9ed8-4a39-8111-46245f166868 admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:03:28.670759 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4afc674e-903e-46c7-a84e-159b181a66b0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:03:28.671027 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4afc674e-903e-46c7-a84e-159b181a66b0 admin admin] PATH_INFO: `/v3/users` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:03:28.671645 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4afc674e-903e-46c7-a84e-159b181a66b0 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:03:28.673580 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4afc674e-903e-46c7-a84e-159b181a66b0 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:03:28.822782 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1071/2128] 10.222.0.27 () {60 vars in 1144 bytes} [Tue Aug 30 02:03:28 2022] POST /identity/v3/users => generated 286 bytes in 219 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 30 02:03:28.822968 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1058/2129] 10.222.0.27 () {60 vars in 1483 bytes} [Tue Aug 30 02:03:28 2022] PUT /identity/v3/projects/14567ca5cc3249f4aef186f05fad1cda/users/b3b5830806e643a1a7577a11a66c554d/roles/1fb81afcb9f245f3b989add94598a063 => generated 0 bytes in 241 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 02:03:28.823065 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4afc674e-903e-46c7-a84e-159b181a66b0 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:03:28.828386 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-4afc674e-903e-46c7-a84e-159b181a66b0 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:03:28.835593 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4afc674e-903e-46c7-a84e-159b181a66b0 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:03:28.840436 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-367f41d8-9ed8-4a39-8111-46245f166868 admin admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:03:28.893171 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d1b0dd7-4268-4133-9f71-b83b88a50683 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:03:28.893906 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d1b0dd7-4268-4133-9f71-b83b88a50683 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:03:28.894184 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d1b0dd7-4268-4133-9f71-b83b88a50683 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:03:28.894479 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d1b0dd7-4268-4133-9f71-b83b88a50683 admin admin] PATH_INFO: `/v3/projects/90b501c6b7e5479a874b5b6cdfaeadb4` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:03:28.895564 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d1b0dd7-4268-4133-9f71-b83b88a50683 admin admin] RBAC: Authorizing `identity:delete_project(project_id=90b501c6b7e5479a874b5b6cdfaeadb4)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:03:28.897635 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d1b0dd7-4268-4133-9f71-b83b88a50683 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:03:28.905069 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-367f41d8-9ed8-4a39-8111-46245f166868 admin admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:03:28.909923 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca565e4b-5175-47a3-ab43-315bc4519311 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:03:28.910607 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca565e4b-5175-47a3-ab43-315bc4519311 admin admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:03:28.910910 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca565e4b-5175-47a3-ab43-315bc4519311 admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:03:28.911180 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca565e4b-5175-47a3-ab43-315bc4519311 admin admin] PATH_INFO: `/v3/roles` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:03:28.911821 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca565e4b-5175-47a3-ab43-315bc4519311 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:03:28.913503 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca565e4b-5175-47a3-ab43-315bc4519311 admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:03:28.924357 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1059/2130] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Aug 30 02:03:28 2022] GET /identity/v3/roles => generated 1420 bytes in 84 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 30 02:03:28.926896 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbed82cf-1d77-4552-a0e6-e73e7d599253 None None] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:03:28.927467 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbed82cf-1d77-4552-a0e6-e73e7d599253 None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:03:28.927632 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbed82cf-1d77-4552-a0e6-e73e7d599253 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:03:28.955705 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.auth.core [None req-cbed82cf-1d77-4552-a0e6-e73e7d599253 None None] MFA Rules not processed for user `b3b5830806e643a1a7577a11a66c554d`. Rule list: `[]` (Enabled: `True`). {{(pid=81578) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:03:28.956889 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-cbed82cf-1d77-4552-a0e6-e73e7d599253 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:03:28.964211 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:03:29.023367 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1060/2131] 10.222.0.27 () {58 vars in 958 bytes} [Tue Aug 30 02:03:28 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 02:03:29.023367 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbed82cf-1d77-4552-a0e6-e73e7d599253 None None] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:03:29.031399 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbed82cf-1d77-4552-a0e6-e73e7d599253 None None] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:03:29.035927 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce3eb874-9f18-44da-8dcb-974d30f1af45 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:03:29.036529 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce3eb874-9f18-44da-8dcb-974d30f1af45 admin admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:03:29.036803 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce3eb874-9f18-44da-8dcb-974d30f1af45 admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:03:29.037074 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce3eb874-9f18-44da-8dcb-974d30f1af45 admin admin] PATH_INFO: `/v3/roles` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:03:29.037748 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce3eb874-9f18-44da-8dcb-974d30f1af45 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:03:29.039316 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce3eb874-9f18-44da-8dcb-974d30f1af45 admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:03:29.049614 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1061/2132] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Aug 30 02:03:29 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 02:03:29.050416 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce3eb874-9f18-44da-8dcb-974d30f1af45 admin admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:03:29.052052 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-ce3eb874-9f18-44da-8dcb-974d30f1af45 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:03:29.079712 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-9d1b0dd7-4268-4133-9f71-b83b88a50683 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '90b501c6b7e5479a874b5b6cdfaeadb4', 'request_id': 'req-9d1b0dd7-4268-4133-9f71-b83b88a50683'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:03:29.085299 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-9d1b0dd7-4268-4133-9f71-b83b88a50683 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '90b501c6b7e5479a874b5b6cdfaeadb4', 'request_id': 'req-9d1b0dd7-4268-4133-9f71-b83b88a50683'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:03:29.094012 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-9d1b0dd7-4268-4133-9f71-b83b88a50683 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '90b501c6b7e5479a874b5b6cdfaeadb4', 'request_id': 'req-9d1b0dd7-4268-4133-9f71-b83b88a50683'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:03:29.114866 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce3eb874-9f18-44da-8dcb-974d30f1af45 admin admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:03:29.130113 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:03:29.165410 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2222a8cd-ca8c-4d8c-8e76-cab14f8d62ff service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ab936b3767a8417aa80c8c8f1b84b667', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4562f3951fbe465280073e60a11e44e2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:03:29.166031 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2222a8cd-ca8c-4d8c-8e76-cab14f8d62ff service cinder] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:03:29.166310 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2222a8cd-ca8c-4d8c-8e76-cab14f8d62ff service cinder] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:03:29.166578 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2222a8cd-ca8c-4d8c-8e76-cab14f8d62ff service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:03:29.221763 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2222a8cd-ca8c-4d8c-8e76-cab14f8d62ff service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:03:29.223176 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2222a8cd-ca8c-4d8c-8e76-cab14f8d62ff service cinder] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:03:29.231994 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1062/2133] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Aug 30 02:03:29 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 182 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 02:03:29.232788 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2222a8cd-ca8c-4d8c-8e76-cab14f8d62ff service cinder] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:03:29.261978 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-9d1b0dd7-4268-4133-9f71-b83b88a50683 admin admin] The token cache is being invalidate because project 90b501c6b7e5479a874b5b6cdfaeadb4 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81577) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 30 02:03:29.262274 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-9d1b0dd7-4268-4133-9f71-b83b88a50683 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:03:29.268266 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-9d1b0dd7-4268-4133-9f71-b83b88a50683 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:03:29.268266 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1072/2134] 10.222.0.27 () {58 vars in 1233 bytes} [Tue Aug 30 02:03:28 2022] DELETE /identity/v3/projects/90b501c6b7e5479a874b5b6cdfaeadb4 => generated 0 bytes in 443 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 02:03:29.269495 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1f1852a-137b-4f6a-8c55-b1657c7f62be None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:03:29.269778 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1f1852a-137b-4f6a-8c55-b1657c7f62be None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:03:29.270087 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1f1852a-137b-4f6a-8c55-b1657c7f62be None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:03:29.291838 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-a1f1852a-137b-4f6a-8c55-b1657c7f62be None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:03:29.293319 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-a1f1852a-137b-4f6a-8c55-b1657c7f62be None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:03:29.302558 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1073/2135] 10.222.0.27 () {58 vars in 958 bytes} [Tue Aug 30 02:03:29 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 30 02:03:29.307160 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2222a8cd-ca8c-4d8c-8e76-cab14f8d62ff service cinder] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:03:29.310682 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62736f09-bd53-4af6-ac63-80239f3520e8 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:03:29.311069 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62736f09-bd53-4af6-ac63-80239f3520e8 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:03:29.311419 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62736f09-bd53-4af6-ac63-80239f3520e8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:03:29.313934 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bd153d5-4294-4e4a-afbe-7a86bc516a1c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:03:29.314968 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bd153d5-4294-4e4a-afbe-7a86bc516a1c admin admin] REQUEST_METHOD: `PUT` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:03:29.315525 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bd153d5-4294-4e4a-afbe-7a86bc516a1c admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:03:29.315801 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bd153d5-4294-4e4a-afbe-7a86bc516a1c admin admin] PATH_INFO: `/v3/projects/f91bad8afa8a44d4ae6a4d31d8f5bfdf/users/de5de5bbd1004b2d83160ef0268b1ac2/roles/1fb81afcb9f245f3b989add94598a063` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:03:29.332121 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bd153d5-4294-4e4a-afbe-7a86bc516a1c admin admin] RBAC: Authorizing `identity:create_grant(project_id=f91bad8afa8a44d4ae6a4d31d8f5bfdf, user_id=de5de5bbd1004b2d83160ef0268b1ac2, role_id=1fb81afcb9f245f3b989add94598a063)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:03:29.332619 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-62736f09-bd53-4af6-ac63-80239f3520e8 None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:03:29.333964 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-62736f09-bd53-4af6-ac63-80239f3520e8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:03:29.334826 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bd153d5-4294-4e4a-afbe-7a86bc516a1c admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:03:29.342886 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1074/2136] 10.222.0.27 () {58 vars in 958 bytes} [Tue Aug 30 02:03:29 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 02:03:29.346963 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62736f09-bd53-4af6-ac63-80239f3520e8 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:03:29.356288 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62736f09-bd53-4af6-ac63-80239f3520e8 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:03:29.362050 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d8f0528-f38c-4bf0-b6c8-9893bff74f9f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:03:29.362651 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d8f0528-f38c-4bf0-b6c8-9893bff74f9f admin admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:03:29.362929 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d8f0528-f38c-4bf0-b6c8-9893bff74f9f admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:03:29.363194 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d8f0528-f38c-4bf0-b6c8-9893bff74f9f admin admin] PATH_INFO: `/v3/domains` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:03:29.363918 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d8f0528-f38c-4bf0-b6c8-9893bff74f9f admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 02:03:29.364219 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d8f0528-f38c-4bf0-b6c8-9893bff74f9f admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:03:29.365859 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d8f0528-f38c-4bf0-b6c8-9893bff74f9f admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:03:29.380123 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1075/2137] 10.222.0.27 () {58 vars in 1153 bytes} [Tue Aug 30 02:03:29 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 02:03:29.391789 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d8f0528-f38c-4bf0-b6c8-9893bff74f9f admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:03:29.401972 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d8f0528-f38c-4bf0-b6c8-9893bff74f9f admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:03:29.407060 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3925e400-1032-4755-829e-84a34df97948 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:03:29.407666 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3925e400-1032-4755-829e-84a34df97948 admin admin] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:03:29.408052 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3925e400-1032-4755-829e-84a34df97948 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:03:29.408366 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3925e400-1032-4755-829e-84a34df97948 admin admin] PATH_INFO: `/v3/projects` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:03:29.408953 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3925e400-1032-4755-829e-84a34df97948 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:03:29.410875 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3925e400-1032-4755-829e-84a34df97948 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:03:29.478760 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1063/2138] 10.222.0.27 () {60 vars in 1483 bytes} [Tue Aug 30 02:03:29 2022] PUT /identity/v3/projects/f91bad8afa8a44d4ae6a4d31d8f5bfdf/users/de5de5bbd1004b2d83160ef0268b1ac2/roles/1fb81afcb9f245f3b989add94598a063 => generated 0 bytes in 246 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 02:03:29.483919 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-412ef1b6-338a-4d2e-b75d-dc0f8bdc937d None None] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:03:29.484219 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-412ef1b6-338a-4d2e-b75d-dc0f8bdc937d None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:03:29.484628 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-412ef1b6-338a-4d2e-b75d-dc0f8bdc937d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:03:29.504660 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.auth.core [None req-412ef1b6-338a-4d2e-b75d-dc0f8bdc937d None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81578) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:03:29.505988 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-412ef1b6-338a-4d2e-b75d-dc0f8bdc937d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:03:29.565671 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1064/2139] 10.222.0.27 () {58 vars in 958 bytes} [Tue Aug 30 02:03:29 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 02:03:29.575991 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-412ef1b6-338a-4d2e-b75d-dc0f8bdc937d None None] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:03:29.577382 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-412ef1b6-338a-4d2e-b75d-dc0f8bdc937d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:03:29.589298 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1076/2140] 10.222.0.27 () {60 vars in 1153 bytes} [Tue Aug 30 02:03:29 2022] POST /identity/v3/projects => generated 405 bytes in 197 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 30 02:03:29.593892 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3925e400-1032-4755-829e-84a34df97948 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:03:29.637713 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-412ef1b6-338a-4d2e-b75d-dc0f8bdc937d None None] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:03:29.659133 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3925e400-1032-4755-829e-84a34df97948 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:03:29.663805 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8ba888b-2887-4cc1-a5e2-0db6986e3206 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:03:29.664406 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8ba888b-2887-4cc1-a5e2-0db6986e3206 admin admin] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:03:29.665314 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8ba888b-2887-4cc1-a5e2-0db6986e3206 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:03:29.665910 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8ba888b-2887-4cc1-a5e2-0db6986e3206 admin admin] PATH_INFO: `/v3/users` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:03:29.666250 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8ba888b-2887-4cc1-a5e2-0db6986e3206 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:03:29.668439 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8ba888b-2887-4cc1-a5e2-0db6986e3206 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:03:29.690327 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdb48f57-798f-4b69-a533-8209deafdd56 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f3ea1580c56f4814a92bbb2acf542905', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4562f3951fbe465280073e60a11e44e2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:03:29.690965 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdb48f57-798f-4b69-a533-8209deafdd56 service neutron] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:03:29.691250 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdb48f57-798f-4b69-a533-8209deafdd56 service neutron] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:03:29.691532 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdb48f57-798f-4b69-a533-8209deafdd56 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:03:29.701194 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bdb48f57-798f-4b69-a533-8209deafdd56 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:03:29.702738 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bdb48f57-798f-4b69-a533-8209deafdd56 service neutron] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:03:29.715041 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1065/2141] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Aug 30 02:03:29 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 140 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 30 02:03:29.839677 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1077/2142] 10.222.0.27 () {60 vars in 1144 bytes} [Tue Aug 30 02:03:29 2022] POST /identity/v3/users => generated 303 bytes in 246 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 30 02:03:29.840384 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdb48f57-798f-4b69-a533-8209deafdd56 service neutron] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:03:29.850804 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdb48f57-798f-4b69-a533-8209deafdd56 service neutron] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:03:29.856048 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-654d926e-dec5-44ae-83fe-ab43552da9da admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:03:29.856656 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-654d926e-dec5-44ae-83fe-ab43552da9da admin admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:03:29.856940 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-654d926e-dec5-44ae-83fe-ab43552da9da admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:03:29.857186 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-654d926e-dec5-44ae-83fe-ab43552da9da admin admin] PATH_INFO: `/v3/roles` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:03:29.857861 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-654d926e-dec5-44ae-83fe-ab43552da9da admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:03:29.859562 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-654d926e-dec5-44ae-83fe-ab43552da9da admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:03:29.872540 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1066/2143] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Aug 30 02:03:29 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 30 02:03:29.877945 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8ba888b-2887-4cc1-a5e2-0db6986e3206 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:03:29.889159 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8ba888b-2887-4cc1-a5e2-0db6986e3206 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:03:29.894466 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c4b4b2b-bc23-4ce3-beff-9e5278b50b74 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:03:29.894679 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c4b4b2b-bc23-4ce3-beff-9e5278b50b74 admin admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:03:29.894962 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c4b4b2b-bc23-4ce3-beff-9e5278b50b74 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:03:29.895235 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c4b4b2b-bc23-4ce3-beff-9e5278b50b74 admin admin] PATH_INFO: `/v3/roles` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:03:29.895894 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c4b4b2b-bc23-4ce3-beff-9e5278b50b74 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:03:29.897558 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c4b4b2b-bc23-4ce3-beff-9e5278b50b74 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:03:29.908799 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1078/2144] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Aug 30 02:03:29 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 02:03:29.913243 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-654d926e-dec5-44ae-83fe-ab43552da9da admin admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:03:29.924455 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-654d926e-dec5-44ae-83fe-ab43552da9da admin admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:03:29.931113 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1089b1d-eacb-4dfb-bdca-fe16a8e7fc48 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:03:29.931967 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1089b1d-eacb-4dfb-bdca-fe16a8e7fc48 admin admin] REQUEST_METHOD: `PUT` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:03:29.932343 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1089b1d-eacb-4dfb-bdca-fe16a8e7fc48 admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:03:29.932757 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1089b1d-eacb-4dfb-bdca-fe16a8e7fc48 admin admin] PATH_INFO: `/v3/projects/4ef7d6a2ea7d44088183599c727b3c14/users/577a7064e6ff47779f830716d90321e7/roles/1fb81afcb9f245f3b989add94598a063` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:03:29.950089 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1089b1d-eacb-4dfb-bdca-fe16a8e7fc48 admin admin] RBAC: Authorizing `identity:create_grant(project_id=4ef7d6a2ea7d44088183599c727b3c14, user_id=577a7064e6ff47779f830716d90321e7, role_id=1fb81afcb9f245f3b989add94598a063)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:03:29.952746 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1089b1d-eacb-4dfb-bdca-fe16a8e7fc48 admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:03:30.126313 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1067/2145] 10.222.0.27 () {60 vars in 1483 bytes} [Tue Aug 30 02:03:29 2022] PUT /identity/v3/projects/4ef7d6a2ea7d44088183599c727b3c14/users/577a7064e6ff47779f830716d90321e7/roles/1fb81afcb9f245f3b989add94598a063 => generated 0 bytes in 213 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 30 02:03:30.131313 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-755faaf1-2fe7-40b2-8ebe-2bfdeb4ca420 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:03:30.131984 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-755faaf1-2fe7-40b2-8ebe-2bfdeb4ca420 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:03:30.132496 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-755faaf1-2fe7-40b2-8ebe-2bfdeb4ca420 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:03:30.165009 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-755faaf1-2fe7-40b2-8ebe-2bfdeb4ca420 None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:03:30.166280 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-755faaf1-2fe7-40b2-8ebe-2bfdeb4ca420 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:03:30.242675 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1079/2146] 10.222.0.27 () {58 vars in 958 bytes} [Tue Aug 30 02:03:30 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 112 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 30 02:03:30.258235 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1089b1d-eacb-4dfb-bdca-fe16a8e7fc48 admin admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:03:30.316244 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1089b1d-eacb-4dfb-bdca-fe16a8e7fc48 admin admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:03:30.319849 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24d07940-b72e-4363-bc22-19dc0da90c08 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f3ea1580c56f4814a92bbb2acf542905', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4562f3951fbe465280073e60a11e44e2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:03:30.320449 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24d07940-b72e-4363-bc22-19dc0da90c08 service neutron] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:03:30.320750 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24d07940-b72e-4363-bc22-19dc0da90c08 service neutron] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:03:30.321019 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24d07940-b72e-4363-bc22-19dc0da90c08 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:03:30.330066 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24d07940-b72e-4363-bc22-19dc0da90c08 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:03:30.331492 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24d07940-b72e-4363-bc22-19dc0da90c08 service neutron] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:03:30.341228 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1068/2147] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Aug 30 02:03:30 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 83 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 02:03:31.210196 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:03:31.721701 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a92ba40-ab81-4218-8cc7-4f7800418918 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:03:31.721701 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a92ba40-ab81-4218-8cc7-4f7800418918 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:03:31.721701 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a92ba40-ab81-4218-8cc7-4f7800418918 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:03:31.754211 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-7a92ba40-ab81-4218-8cc7-4f7800418918 None None] MFA Rules not processed for user `de5de5bbd1004b2d83160ef0268b1ac2`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:03:31.755422 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-7a92ba40-ab81-4218-8cc7-4f7800418918 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:03:31.829606 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1080/2148] 10.222.0.27 () {58 vars in 958 bytes} [Tue Aug 30 02:03:31 2022] POST /identity/v3/auth/tokens => generated 3248 bytes in 111 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 02:03:31.830702 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24d07940-b72e-4363-bc22-19dc0da90c08 service neutron] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:03:31.832128 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-24d07940-b72e-4363-bc22-19dc0da90c08 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:03:31.845213 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24d07940-b72e-4363-bc22-19dc0da90c08 service neutron] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:03:31.849787 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0145979a-18cc-4ba4-8189-bd187cdb7990 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:03:31.850397 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0145979a-18cc-4ba4-8189-bd187cdb7990 admin admin] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:03:31.850677 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0145979a-18cc-4ba4-8189-bd187cdb7990 admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:03:31.850945 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0145979a-18cc-4ba4-8189-bd187cdb7990 admin admin] PATH_INFO: `/v3/projects` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:03:31.851581 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0145979a-18cc-4ba4-8189-bd187cdb7990 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:03:31.854103 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0145979a-18cc-4ba4-8189-bd187cdb7990 admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:03:31.945209 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1069/2149] 10.222.0.27 () {60 vars in 1153 bytes} [Tue Aug 30 02:03:31 2022] POST /identity/v3/projects => generated 369 bytes in 112 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 30 02:03:31.946801 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a92ba40-ab81-4218-8cc7-4f7800418918 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:03:31.958543 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a92ba40-ab81-4218-8cc7-4f7800418918 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:03:32.018195 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a4c03a9-d952-43c5-92fe-99e81c105fc7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:03:32.018911 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a4c03a9-d952-43c5-92fe-99e81c105fc7 admin admin] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:03:32.019293 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a4c03a9-d952-43c5-92fe-99e81c105fc7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:03:32.019693 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a4c03a9-d952-43c5-92fe-99e81c105fc7 admin admin] PATH_INFO: `/v3/users` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:03:32.020430 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a4c03a9-d952-43c5-92fe-99e81c105fc7 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:03:32.022602 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a4c03a9-d952-43c5-92fe-99e81c105fc7 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:03:32.258812 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1081/2150] 10.222.0.27 () {60 vars in 1144 bytes} [Tue Aug 30 02:03:31 2022] POST /identity/v3/users => generated 291 bytes in 312 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 30 02:03:32.262365 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0145979a-18cc-4ba4-8189-bd187cdb7990 admin admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:03:32.262804 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a3d5caf-8c2c-4f03-9e68-0ab3d2973097 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:03:32.263220 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a3d5caf-8c2c-4f03-9e68-0ab3d2973097 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:03:32.263509 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a3d5caf-8c2c-4f03-9e68-0ab3d2973097 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:03:32.275172 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0145979a-18cc-4ba4-8189-bd187cdb7990 admin admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:03:32.281294 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46c193c4-4721-4789-8b5d-767b1d1e5307 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:03:32.282314 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46c193c4-4721-4789-8b5d-767b1d1e5307 admin admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:03:32.282716 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46c193c4-4721-4789-8b5d-767b1d1e5307 admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:03:32.283103 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46c193c4-4721-4789-8b5d-767b1d1e5307 admin admin] PATH_INFO: `/v3/roles` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:03:32.283949 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46c193c4-4721-4789-8b5d-767b1d1e5307 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:03:32.285722 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46c193c4-4721-4789-8b5d-767b1d1e5307 admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:03:32.293943 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-4a3d5caf-8c2c-4f03-9e68-0ab3d2973097 None None] MFA Rules not processed for user `577a7064e6ff47779f830716d90321e7`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:03:32.295159 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-4a3d5caf-8c2c-4f03-9e68-0ab3d2973097 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:03:32.297732 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1070/2151] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Aug 30 02:03:32 2022] GET /identity/v3/roles => generated 1420 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 02:03:32.301786 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46c193c4-4721-4789-8b5d-767b1d1e5307 admin admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:03:32.313261 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46c193c4-4721-4789-8b5d-767b1d1e5307 admin admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:03:32.318716 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b634941-e0fd-4bce-bbf6-09f6463952a5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:03:32.319366 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b634941-e0fd-4bce-bbf6-09f6463952a5 admin admin] REQUEST_METHOD: `PUT` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:03:32.319642 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b634941-e0fd-4bce-bbf6-09f6463952a5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:03:32.319988 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b634941-e0fd-4bce-bbf6-09f6463952a5 admin admin] PATH_INFO: `/v3/projects/c4ae66af82a54303bcfb9c38a7081688/users/f865d9c0901849269956da4aad9a7d88/roles/ead5fac86a0747ea8823a82efcea8ed7` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:03:32.335279 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b634941-e0fd-4bce-bbf6-09f6463952a5 admin admin] RBAC: Authorizing `identity:create_grant(project_id=c4ae66af82a54303bcfb9c38a7081688, user_id=f865d9c0901849269956da4aad9a7d88, role_id=ead5fac86a0747ea8823a82efcea8ed7)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:03:32.338154 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b634941-e0fd-4bce-bbf6-09f6463952a5 admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:03:32.363913 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1082/2152] 10.222.0.27 () {58 vars in 958 bytes} [Tue Aug 30 02:03:32 2022] POST /identity/v3/auth/tokens => generated 3282 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 02:03:32.368854 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a3d5caf-8c2c-4f03-9e68-0ab3d2973097 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:03:32.379094 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a3d5caf-8c2c-4f03-9e68-0ab3d2973097 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:03:32.384620 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7c9200c-00dc-463c-be5c-787cb2e35f3c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:03:32.385207 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7c9200c-00dc-463c-be5c-787cb2e35f3c admin admin] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:03:32.385509 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7c9200c-00dc-463c-be5c-787cb2e35f3c admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:03:32.385779 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7c9200c-00dc-463c-be5c-787cb2e35f3c admin admin] PATH_INFO: `/v3/projects` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:03:32.386409 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7c9200c-00dc-463c-be5c-787cb2e35f3c admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:03:32.388307 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7c9200c-00dc-463c-be5c-787cb2e35f3c admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:03:32.399174 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:DELETE=11,keystone:SELECT=88,keystone:INSERT=1 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:03:32.433619 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1071/2153] 10.222.0.27 () {60 vars in 1483 bytes} [Tue Aug 30 02:03:32 2022] PUT /identity/v3/projects/c4ae66af82a54303bcfb9c38a7081688/users/f865d9c0901849269956da4aad9a7d88/roles/ead5fac86a0747ea8823a82efcea8ed7 => generated 0 bytes in 132 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 30 02:03:32.437351 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b634941-e0fd-4bce-bbf6-09f6463952a5 admin admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:03:32.463705 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=97,keystone:DELETE=3 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:03:32.501062 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b634941-e0fd-4bce-bbf6-09f6463952a5 admin admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:03:32.505786 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-368a5d7d-98b5-4c0b-aaa8-c9858a949d4a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:03:32.506378 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-368a5d7d-98b5-4c0b-aaa8-c9858a949d4a admin admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:03:32.506654 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-368a5d7d-98b5-4c0b-aaa8-c9858a949d4a admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:03:32.506926 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-368a5d7d-98b5-4c0b-aaa8-c9858a949d4a admin admin] PATH_INFO: `/v3/roles` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:03:32.507548 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-368a5d7d-98b5-4c0b-aaa8-c9858a949d4a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:03:32.509284 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-368a5d7d-98b5-4c0b-aaa8-c9858a949d4a admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:03:32.514052 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1083/2154] 10.222.0.27 () {60 vars in 1153 bytes} [Tue Aug 30 02:03:32 2022] POST /identity/v3/projects => generated 405 bytes in 146 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 30 02:03:32.517999 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7c9200c-00dc-463c-be5c-787cb2e35f3c admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:03:32.522924 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1072/2155] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Aug 30 02:03:32 2022] GET /identity/v3/roles => generated 1420 bytes in 85 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 30 02:03:32.556976 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-368a5d7d-98b5-4c0b-aaa8-c9858a949d4a admin admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:03:32.567091 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-368a5d7d-98b5-4c0b-aaa8-c9858a949d4a admin admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:03:32.581140 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7c9200c-00dc-463c-be5c-787cb2e35f3c admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:03:32.586768 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79a50a93-32d6-47b1-a6ef-f995deb22ebe admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:03:32.593268 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79a50a93-32d6-47b1-a6ef-f995deb22ebe admin admin] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:03:32.593268 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79a50a93-32d6-47b1-a6ef-f995deb22ebe admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:03:32.593268 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79a50a93-32d6-47b1-a6ef-f995deb22ebe admin admin] PATH_INFO: `/v3/users` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:03:32.593268 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79a50a93-32d6-47b1-a6ef-f995deb22ebe admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:03:32.593268 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79a50a93-32d6-47b1-a6ef-f995deb22ebe admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:03:32.625941 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ca27adb-242b-49df-8594-8706c22bed55 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:03:32.626589 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ca27adb-242b-49df-8594-8706c22bed55 admin admin] REQUEST_METHOD: `PUT` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:03:32.626865 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ca27adb-242b-49df-8594-8706c22bed55 admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:03:32.627130 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ca27adb-242b-49df-8594-8706c22bed55 admin admin] PATH_INFO: `/v3/projects/c4ae66af82a54303bcfb9c38a7081688/users/f865d9c0901849269956da4aad9a7d88/roles/ead5fac86a0747ea8823a82efcea8ed7` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:03:32.629015 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ca27adb-242b-49df-8594-8706c22bed55 admin admin] RBAC: Authorizing `identity:create_grant(project_id=c4ae66af82a54303bcfb9c38a7081688, user_id=f865d9c0901849269956da4aad9a7d88, role_id=ead5fac86a0747ea8823a82efcea8ed7)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:03:32.631652 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ca27adb-242b-49df-8594-8706c22bed55 admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:03:32.641534 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1073/2156] 10.222.0.27 () {60 vars in 1483 bytes} [Tue Aug 30 02:03:32 2022] PUT /identity/v3/projects/c4ae66af82a54303bcfb9c38a7081688/users/f865d9c0901849269956da4aad9a7d88/roles/ead5fac86a0747ea8823a82efcea8ed7 => generated 0 bytes in 85 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 30 02:03:32.708307 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1084/2157] 10.222.0.27 () {60 vars in 1144 bytes} [Tue Aug 30 02:03:32 2022] POST /identity/v3/users => generated 309 bytes in 190 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 30 02:03:32.711784 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ca27adb-242b-49df-8594-8706c22bed55 admin admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:03:32.776559 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ca27adb-242b-49df-8594-8706c22bed55 admin admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:03:32.781160 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9c09124-c6e7-46bd-903d-494970419302 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:03:32.781791 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9c09124-c6e7-46bd-903d-494970419302 admin admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:03:32.782072 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9c09124-c6e7-46bd-903d-494970419302 admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:03:32.782430 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9c09124-c6e7-46bd-903d-494970419302 admin admin] PATH_INFO: `/v3/roles` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:03:32.783066 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9c09124-c6e7-46bd-903d-494970419302 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:03:32.784643 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9c09124-c6e7-46bd-903d-494970419302 admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:03:32.795093 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1074/2158] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Aug 30 02:03:32 2022] GET /identity/v3/roles => generated 1420 bytes in 84 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 02:03:32.799117 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79a50a93-32d6-47b1-a6ef-f995deb22ebe admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:03:32.809967 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79a50a93-32d6-47b1-a6ef-f995deb22ebe admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:03:32.815200 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbf72a55-977d-45b6-a908-974d8924c011 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:03:32.818987 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbf72a55-977d-45b6-a908-974d8924c011 admin admin] REQUEST_METHOD: `PUT` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:03:32.818987 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbf72a55-977d-45b6-a908-974d8924c011 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:03:32.818987 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbf72a55-977d-45b6-a908-974d8924c011 admin admin] PATH_INFO: `/v3/projects/21531e42055345df894058be701bd2c3/users/3403b440afe84368a9254cf8cc7c8c13/roles/ead5fac86a0747ea8823a82efcea8ed7` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:03:32.831739 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbf72a55-977d-45b6-a908-974d8924c011 admin admin] RBAC: Authorizing `identity:create_grant(project_id=21531e42055345df894058be701bd2c3, user_id=3403b440afe84368a9254cf8cc7c8c13, role_id=ead5fac86a0747ea8823a82efcea8ed7)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:03:32.834431 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbf72a55-977d-45b6-a908-974d8924c011 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:03:32.941415 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1085/2159] 10.222.0.27 () {60 vars in 1483 bytes} [Tue Aug 30 02:03:32 2022] PUT /identity/v3/projects/21531e42055345df894058be701bd2c3/users/3403b440afe84368a9254cf8cc7c8c13/roles/ead5fac86a0747ea8823a82efcea8ed7 => generated 0 bytes in 143 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 02:03:32.945425 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9c09124-c6e7-46bd-903d-494970419302 admin admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:03:32.955527 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9c09124-c6e7-46bd-903d-494970419302 admin admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:03:33.013654 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-770f5183-b423-417b-8b9a-49cedbd13a97 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:03:33.014266 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-770f5183-b423-417b-8b9a-49cedbd13a97 admin admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:03:33.014543 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-770f5183-b423-417b-8b9a-49cedbd13a97 admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:03:33.014809 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-770f5183-b423-417b-8b9a-49cedbd13a97 admin admin] PATH_INFO: `/v3/roles` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:03:33.015446 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-770f5183-b423-417b-8b9a-49cedbd13a97 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:03:33.017401 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-770f5183-b423-417b-8b9a-49cedbd13a97 admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:03:33.033295 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1075/2160] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Aug 30 02:03:32 2022] GET /identity/v3/roles => generated 1420 bytes in 85 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 02:03:33.034467 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbf72a55-977d-45b6-a908-974d8924c011 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:03:33.046227 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbf72a55-977d-45b6-a908-974d8924c011 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:03:33.056110 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22075d44-a520-47f3-8628-05095380ae10 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:03:33.058321 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22075d44-a520-47f3-8628-05095380ae10 admin admin] REQUEST_METHOD: `PUT` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:03:33.058321 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22075d44-a520-47f3-8628-05095380ae10 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:03:33.058321 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22075d44-a520-47f3-8628-05095380ae10 admin admin] PATH_INFO: `/v3/projects/21531e42055345df894058be701bd2c3/users/3403b440afe84368a9254cf8cc7c8c13/roles/ead5fac86a0747ea8823a82efcea8ed7` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:03:33.059401 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22075d44-a520-47f3-8628-05095380ae10 admin admin] RBAC: Authorizing `identity:create_grant(project_id=21531e42055345df894058be701bd2c3, user_id=3403b440afe84368a9254cf8cc7c8c13, role_id=ead5fac86a0747ea8823a82efcea8ed7)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:03:33.062147 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22075d44-a520-47f3-8628-05095380ae10 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:03:33.073017 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1086/2161] 10.222.0.27 () {60 vars in 1483 bytes} [Tue Aug 30 02:03:33 2022] PUT /identity/v3/projects/21531e42055345df894058be701bd2c3/users/3403b440afe84368a9254cf8cc7c8c13/roles/ead5fac86a0747ea8823a82efcea8ed7 => generated 0 bytes in 39 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 02:03:34.171682 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=84,keystone:DELETE=14,keystone:INSERT=2 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:03:34.438930 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc4943d3-bdae-437e-836a-e926e0efc4ec None None] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:03:34.439458 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc4943d3-bdae-437e-836a-e926e0efc4ec None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:03:34.439739 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc4943d3-bdae-437e-836a-e926e0efc4ec None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:03:34.472162 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.auth.core [None req-bc4943d3-bdae-437e-836a-e926e0efc4ec None None] MFA Rules not processed for user `f865d9c0901849269956da4aad9a7d88`. Rule list: `[]` (Enabled: `True`). {{(pid=81578) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:03:34.475714 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-bc4943d3-bdae-437e-836a-e926e0efc4ec None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:03:34.549484 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1076/2162] 10.222.0.27 () {58 vars in 958 bytes} [Tue Aug 30 02:03:34 2022] POST /identity/v3/auth/tokens => generated 3313 bytes in 111 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 02:03:34.561188 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22075d44-a520-47f3-8628-05095380ae10 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:03:34.563044 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-22075d44-a520-47f3-8628-05095380ae10 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:03:34.595361 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cab75c6d-1364-4fb3-9572-e0050585d11b None None] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:03:34.595720 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cab75c6d-1364-4fb3-9572-e0050585d11b None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:03:34.596057 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cab75c6d-1364-4fb3-9572-e0050585d11b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:03:34.628130 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.auth.core [None req-cab75c6d-1364-4fb3-9572-e0050585d11b None None] MFA Rules not processed for user `3403b440afe84368a9254cf8cc7c8c13`. Rule list: `[]` (Enabled: `True`). {{(pid=81578) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:03:34.629677 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-cab75c6d-1364-4fb3-9572-e0050585d11b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:03:34.631977 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22075d44-a520-47f3-8628-05095380ae10 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:03:34.635993 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-248da203-aa95-47a3-b97a-d2a5381cc6ce service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ab936b3767a8417aa80c8c8f1b84b667', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4562f3951fbe465280073e60a11e44e2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:03:34.636637 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-248da203-aa95-47a3-b97a-d2a5381cc6ce service cinder] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:03:34.636895 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-248da203-aa95-47a3-b97a-d2a5381cc6ce service cinder] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:03:34.637166 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-248da203-aa95-47a3-b97a-d2a5381cc6ce service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:03:34.707079 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-248da203-aa95-47a3-b97a-d2a5381cc6ce service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:03:34.707079 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-248da203-aa95-47a3-b97a-d2a5381cc6ce service cinder] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:03:34.713560 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1077/2163] 10.222.0.27 () {58 vars in 958 bytes} [Tue Aug 30 02:03:34 2022] POST /identity/v3/auth/tokens => generated 3349 bytes in 119 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 02:03:34.724305 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1087/2164] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Aug 30 02:03:34 2022] GET /identity/v3/auth/tokens => generated 3248 bytes in 161 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 02:03:34.895367 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:DELETE=21,keystone:SELECT=77,keystone:INSERT=2 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:03:36.219951 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c803eccb-b576-45f2-9305-8b86628217b8 None None] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:03:36.220451 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c803eccb-b576-45f2-9305-8b86628217b8 None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:03:36.220659 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c803eccb-b576-45f2-9305-8b86628217b8 None None] PATH_INFO: `/` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:03:36.222132 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1078/2165] 10.222.0.27 () {54 vars in 921 bytes} [Tue Aug 30 02:03:36 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 02:03:36.232776 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31ffede7-97ce-400e-83b5-2014a3039e25 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:03:36.233252 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31ffede7-97ce-400e-83b5-2014a3039e25 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:03:36.233342 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31ffede7-97ce-400e-83b5-2014a3039e25 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:03:36.253564 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-31ffede7-97ce-400e-83b5-2014a3039e25 None None] MFA Rules not processed for user `379536ff06fd4f719899e5dde732e63d`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:03:36.254906 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-31ffede7-97ce-400e-83b5-2014a3039e25 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:03:36.331587 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1088/2166] 10.222.0.27 () {58 vars in 1018 bytes} [Tue Aug 30 02:03:36 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 02:03:36.349994 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c803eccb-b576-45f2-9305-8b86628217b8 None None] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:03:36.349994 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-c803eccb-b576-45f2-9305-8b86628217b8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:03:36.367135 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c803eccb-b576-45f2-9305-8b86628217b8 None None] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:03:36.372929 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a880514e-7335-48ce-807a-cb362886ba57 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '379536ff06fd4f719899e5dde732e63d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4562f3951fbe465280073e60a11e44e2', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:03:36.373662 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a880514e-7335-48ce-807a-cb362886ba57 service placement] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:03:36.374015 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a880514e-7335-48ce-807a-cb362886ba57 service placement] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:03:36.374358 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a880514e-7335-48ce-807a-cb362886ba57 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:03:36.383930 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a880514e-7335-48ce-807a-cb362886ba57 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:03:36.385583 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a880514e-7335-48ce-807a-cb362886ba57 service placement] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:03:36.400585 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31ffede7-97ce-400e-83b5-2014a3039e25 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:03:36.410342 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1079/2167] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Aug 30 02:03:36 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 65 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 02:03:36.471043 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31ffede7-97ce-400e-83b5-2014a3039e25 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:03:36.475743 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abd1ecc5-0ea5-4c6c-9c04-25c458f0c2a8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:03:36.476405 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abd1ecc5-0ea5-4c6c-9c04-25c458f0c2a8 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:03:36.476723 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abd1ecc5-0ea5-4c6c-9c04-25c458f0c2a8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:03:36.476990 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abd1ecc5-0ea5-4c6c-9c04-25c458f0c2a8 admin admin] PATH_INFO: `/v3/users/577a7064e6ff47779f830716d90321e7` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:03:36.478139 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abd1ecc5-0ea5-4c6c-9c04-25c458f0c2a8 admin admin] RBAC: Authorizing `identity:delete_user(user_id=577a7064e6ff47779f830716d90321e7)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:03:36.480074 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abd1ecc5-0ea5-4c6c-9c04-25c458f0c2a8 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:03:36.753667 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-abd1ecc5-0ea5-4c6c-9c04-25c458f0c2a8 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '577a7064e6ff47779f830716d90321e7'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:03:36.754336 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-abd1ecc5-0ea5-4c6c-9c04-25c458f0c2a8 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '577a7064e6ff47779f830716d90321e7'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:03:36.754943 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-abd1ecc5-0ea5-4c6c-9c04-25c458f0c2a8 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '577a7064e6ff47779f830716d90321e7'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:03:36.770741 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-abd1ecc5-0ea5-4c6c-9c04-25c458f0c2a8 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '577a7064e6ff47779f830716d90321e7'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:03:36.778406 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-abd1ecc5-0ea5-4c6c-9c04-25c458f0c2a8 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '577a7064e6ff47779f830716d90321e7'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:03:36.889318 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1089/2168] 10.222.0.27 () {58 vars in 1224 bytes} [Tue Aug 30 02:03:36 2022] DELETE /identity/v3/users/577a7064e6ff47779f830716d90321e7 => generated 0 bytes in 490 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 02:03:36.958077 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a880514e-7335-48ce-807a-cb362886ba57 service placement] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:03:36.960127 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-a880514e-7335-48ce-807a-cb362886ba57 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:03:36.972539 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a880514e-7335-48ce-807a-cb362886ba57 service placement] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:03:37.024700 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29129e21-f615-4de1-9214-30b0a0b02133 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ab936b3767a8417aa80c8c8f1b84b667', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4562f3951fbe465280073e60a11e44e2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:03:37.025311 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29129e21-f615-4de1-9214-30b0a0b02133 service cinder] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:03:37.025623 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29129e21-f615-4de1-9214-30b0a0b02133 service cinder] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:03:37.025898 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29129e21-f615-4de1-9214-30b0a0b02133 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:03:37.027652 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-29129e21-f615-4de1-9214-30b0a0b02133 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:03:37.093655 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29129e21-f615-4de1-9214-30b0a0b02133 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:03:37.095108 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29129e21-f615-4de1-9214-30b0a0b02133 service cinder] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:03:37.105800 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1080/2169] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Aug 30 02:03:36 2022] GET /identity/v3/auth/tokens => generated 3313 bytes in 149 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 02:03:37.358971 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abd1ecc5-0ea5-4c6c-9c04-25c458f0c2a8 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:03:37.358971 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-abd1ecc5-0ea5-4c6c-9c04-25c458f0c2a8 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:03:37.453660 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abd1ecc5-0ea5-4c6c-9c04-25c458f0c2a8 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:03:37.459127 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75f30914-d4c2-4873-a614-dbaf4caa66b1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:03:37.459789 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75f30914-d4c2-4873-a614-dbaf4caa66b1 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:03:37.460073 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75f30914-d4c2-4873-a614-dbaf4caa66b1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:03:37.460429 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75f30914-d4c2-4873-a614-dbaf4caa66b1 admin admin] PATH_INFO: `/v3/projects/4ef7d6a2ea7d44088183599c727b3c14` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:03:37.462295 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75f30914-d4c2-4873-a614-dbaf4caa66b1 admin admin] RBAC: Authorizing `identity:delete_project(project_id=4ef7d6a2ea7d44088183599c727b3c14)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:03:37.464266 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75f30914-d4c2-4873-a614-dbaf4caa66b1 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:03:37.628976 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-75f30914-d4c2-4873-a614-dbaf4caa66b1 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '4ef7d6a2ea7d44088183599c727b3c14', 'request_id': 'req-75f30914-d4c2-4873-a614-dbaf4caa66b1'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:03:37.634874 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-75f30914-d4c2-4873-a614-dbaf4caa66b1 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '4ef7d6a2ea7d44088183599c727b3c14', 'request_id': 'req-75f30914-d4c2-4873-a614-dbaf4caa66b1'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:03:37.641263 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-75f30914-d4c2-4873-a614-dbaf4caa66b1 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '4ef7d6a2ea7d44088183599c727b3c14', 'request_id': 'req-75f30914-d4c2-4873-a614-dbaf4caa66b1'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:03:37.827094 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=96,keystone:DELETE=4 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:03:37.833164 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-75f30914-d4c2-4873-a614-dbaf4caa66b1 admin admin] The token cache is being invalidate because project 4ef7d6a2ea7d44088183599c727b3c14 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81577) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 30 02:03:37.833634 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-75f30914-d4c2-4873-a614-dbaf4caa66b1 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:03:37.834268 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-75f30914-d4c2-4873-a614-dbaf4caa66b1 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:03:37.837807 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1090/2170] 10.222.0.27 () {58 vars in 1233 bytes} [Tue Aug 30 02:03:37 2022] DELETE /identity/v3/projects/4ef7d6a2ea7d44088183599c727b3c14 => generated 0 bytes in 481 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 02:03:37.840729 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29129e21-f615-4de1-9214-30b0a0b02133 service cinder] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:03:37.863374 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29129e21-f615-4de1-9214-30b0a0b02133 service cinder] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:03:37.937803 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a57b6cf4-3da3-49a1-bfb8-b92c40ee723d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:03:37.938445 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a57b6cf4-3da3-49a1-bfb8-b92c40ee723d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:03:37.938855 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a57b6cf4-3da3-49a1-bfb8-b92c40ee723d admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:03:37.939135 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a57b6cf4-3da3-49a1-bfb8-b92c40ee723d admin admin] PATH_INFO: `/v3/users/3403b440afe84368a9254cf8cc7c8c13` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:03:37.940261 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a57b6cf4-3da3-49a1-bfb8-b92c40ee723d admin admin] RBAC: Authorizing `identity:delete_user(user_id=3403b440afe84368a9254cf8cc7c8c13)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:03:37.942286 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a57b6cf4-3da3-49a1-bfb8-b92c40ee723d admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:03:38.263107 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-a57b6cf4-3da3-49a1-bfb8-b92c40ee723d admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '3403b440afe84368a9254cf8cc7c8c13'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:03:38.263749 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-a57b6cf4-3da3-49a1-bfb8-b92c40ee723d admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '3403b440afe84368a9254cf8cc7c8c13'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:03:38.414938 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-a57b6cf4-3da3-49a1-bfb8-b92c40ee723d admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '3403b440afe84368a9254cf8cc7c8c13'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:03:38.415575 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-a57b6cf4-3da3-49a1-bfb8-b92c40ee723d admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '3403b440afe84368a9254cf8cc7c8c13'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:03:38.422837 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-a57b6cf4-3da3-49a1-bfb8-b92c40ee723d admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '3403b440afe84368a9254cf8cc7c8c13'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:03:38.440163 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1081/2171] 10.222.0.27 () {58 vars in 1224 bytes} [Tue Aug 30 02:03:37 2022] DELETE /identity/v3/users/3403b440afe84368a9254cf8cc7c8c13 => generated 0 bytes in 600 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 02:03:38.758116 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75f30914-d4c2-4873-a614-dbaf4caa66b1 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:03:38.758116 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-75f30914-d4c2-4873-a614-dbaf4caa66b1 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:03:38.771099 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75f30914-d4c2-4873-a614-dbaf4caa66b1 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:03:38.830193 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1435e33-5115-4177-8239-6e140208a9fe admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:03:38.830829 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1435e33-5115-4177-8239-6e140208a9fe admin admin] REQUEST_METHOD: `DELETE` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:03:38.831110 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1435e33-5115-4177-8239-6e140208a9fe admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:03:38.831384 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1435e33-5115-4177-8239-6e140208a9fe admin admin] PATH_INFO: `/v3/projects/21531e42055345df894058be701bd2c3` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:03:38.832495 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1435e33-5115-4177-8239-6e140208a9fe admin admin] RBAC: Authorizing `identity:delete_project(project_id=21531e42055345df894058be701bd2c3)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:03:38.834467 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1435e33-5115-4177-8239-6e140208a9fe admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:03:38.975731 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-e1435e33-5115-4177-8239-6e140208a9fe admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '21531e42055345df894058be701bd2c3', 'request_id': 'req-e1435e33-5115-4177-8239-6e140208a9fe'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:03:38.982257 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-e1435e33-5115-4177-8239-6e140208a9fe admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '21531e42055345df894058be701bd2c3', 'request_id': 'req-e1435e33-5115-4177-8239-6e140208a9fe'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:03:38.988452 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-e1435e33-5115-4177-8239-6e140208a9fe admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '21531e42055345df894058be701bd2c3', 'request_id': 'req-e1435e33-5115-4177-8239-6e140208a9fe'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:03:38.995665 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=86,keystone:DELETE=12,keystone:INSERT=2 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:03:39.072004 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-e1435e33-5115-4177-8239-6e140208a9fe admin admin] The token cache is being invalidate because project 21531e42055345df894058be701bd2c3 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81577) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 30 02:03:39.072206 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-e1435e33-5115-4177-8239-6e140208a9fe admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:03:39.072903 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-e1435e33-5115-4177-8239-6e140208a9fe admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:03:39.074423 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1091/2172] 10.222.0.27 () {58 vars in 1233 bytes} [Tue Aug 30 02:03:38 2022] DELETE /identity/v3/projects/21531e42055345df894058be701bd2c3 => generated 0 bytes in 321 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 30 02:03:39.079437 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-345d7f4c-a7aa-4118-b479-5ddd6ce67d1c None None] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:03:39.079898 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-345d7f4c-a7aa-4118-b479-5ddd6ce67d1c None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:03:39.079987 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-345d7f4c-a7aa-4118-b479-5ddd6ce67d1c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:03:39.100712 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.auth.core [None req-345d7f4c-a7aa-4118-b479-5ddd6ce67d1c None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81578) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:03:39.102566 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-345d7f4c-a7aa-4118-b479-5ddd6ce67d1c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:03:39.162946 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1082/2173] 10.222.0.27 () {58 vars in 958 bytes} [Tue Aug 30 02:03:39 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 02:03:39.171642 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60390dee-1cb5-461c-9976-2da0708c1049 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:03:39.171933 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60390dee-1cb5-461c-9976-2da0708c1049 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:03:39.172211 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60390dee-1cb5-461c-9976-2da0708c1049 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:03:39.192239 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-60390dee-1cb5-461c-9976-2da0708c1049 None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:03:39.193432 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-60390dee-1cb5-461c-9976-2da0708c1049 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:03:39.201968 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1092/2174] 10.222.0.27 () {58 vars in 958 bytes} [Tue Aug 30 02:03:39 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 02:03:39.206180 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-345d7f4c-a7aa-4118-b479-5ddd6ce67d1c None None] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:03:39.216959 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-345d7f4c-a7aa-4118-b479-5ddd6ce67d1c None None] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:03:39.221502 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bad219b9-c175-4cd2-8ca5-72271b0f8371 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:03:39.222105 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bad219b9-c175-4cd2-8ca5-72271b0f8371 admin admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:03:39.222399 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bad219b9-c175-4cd2-8ca5-72271b0f8371 admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:03:39.222668 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bad219b9-c175-4cd2-8ca5-72271b0f8371 admin admin] PATH_INFO: `/v3/domains` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:03:39.223315 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bad219b9-c175-4cd2-8ca5-72271b0f8371 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81578) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 02:03:39.223613 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bad219b9-c175-4cd2-8ca5-72271b0f8371 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:03:39.225228 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bad219b9-c175-4cd2-8ca5-72271b0f8371 admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:03:39.236452 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1083/2175] 10.222.0.27 () {58 vars in 1153 bytes} [Tue Aug 30 02:03:39 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 02:03:39.240703 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60390dee-1cb5-461c-9976-2da0708c1049 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:03:39.250033 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60390dee-1cb5-461c-9976-2da0708c1049 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:03:39.254930 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f92545e7-def7-4f9c-8eb3-a06404bd4bc6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:03:39.259373 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f92545e7-def7-4f9c-8eb3-a06404bd4bc6 admin admin] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:03:39.259373 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f92545e7-def7-4f9c-8eb3-a06404bd4bc6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:03:39.259373 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f92545e7-def7-4f9c-8eb3-a06404bd4bc6 admin admin] PATH_INFO: `/v3/projects` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:03:39.259373 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f92545e7-def7-4f9c-8eb3-a06404bd4bc6 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:03:39.259373 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f92545e7-def7-4f9c-8eb3-a06404bd4bc6 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:03:39.364944 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1093/2176] 10.222.0.27 () {60 vars in 1153 bytes} [Tue Aug 30 02:03:39 2022] POST /identity/v3/projects => generated 389 bytes in 125 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 02:03:39.369498 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bad219b9-c175-4cd2-8ca5-72271b0f8371 admin admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:03:39.379227 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bad219b9-c175-4cd2-8ca5-72271b0f8371 admin admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:03:39.435027 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1e4ead2-296a-4e3c-877b-27c72b35d7d4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:03:39.435629 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1e4ead2-296a-4e3c-877b-27c72b35d7d4 admin admin] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:03:39.435908 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1e4ead2-296a-4e3c-877b-27c72b35d7d4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:03:39.436179 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1e4ead2-296a-4e3c-877b-27c72b35d7d4 admin admin] PATH_INFO: `/v3/users` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:03:39.436818 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1e4ead2-296a-4e3c-877b-27c72b35d7d4 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:03:39.438729 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1e4ead2-296a-4e3c-877b-27c72b35d7d4 admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:03:39.506378 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1084/2177] 10.222.0.27 () {60 vars in 1144 bytes} [Tue Aug 30 02:03:39 2022] POST /identity/v3/users => generated 295 bytes in 137 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 02:03:39.511181 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f92545e7-def7-4f9c-8eb3-a06404bd4bc6 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:03:39.522035 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f92545e7-def7-4f9c-8eb3-a06404bd4bc6 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:03:39.527467 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f370107e-ce4e-4861-b3cb-79f007881cc5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:03:39.528077 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f370107e-ce4e-4861-b3cb-79f007881cc5 admin admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:03:39.528349 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f370107e-ce4e-4861-b3cb-79f007881cc5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:03:39.528765 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f370107e-ce4e-4861-b3cb-79f007881cc5 admin admin] PATH_INFO: `/v3/roles` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:03:39.529719 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f370107e-ce4e-4861-b3cb-79f007881cc5 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:03:39.531633 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f370107e-ce4e-4861-b3cb-79f007881cc5 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:03:39.544764 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1094/2178] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Aug 30 02:03:39 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 02:03:39.546191 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1e4ead2-296a-4e3c-877b-27c72b35d7d4 admin admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:03:39.556586 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1e4ead2-296a-4e3c-877b-27c72b35d7d4 admin admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:03:39.561322 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d88a68aa-115e-4134-a70d-8787dfe38446 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:03:39.562036 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d88a68aa-115e-4134-a70d-8787dfe38446 admin admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:03:39.562312 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d88a68aa-115e-4134-a70d-8787dfe38446 admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:03:39.562581 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d88a68aa-115e-4134-a70d-8787dfe38446 admin admin] PATH_INFO: `/v3/roles` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:03:39.563217 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d88a68aa-115e-4134-a70d-8787dfe38446 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:03:39.564788 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d88a68aa-115e-4134-a70d-8787dfe38446 admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:03:39.575873 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1085/2179] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Aug 30 02:03:39 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 02:03:39.579954 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f370107e-ce4e-4861-b3cb-79f007881cc5 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:03:39.590523 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f370107e-ce4e-4861-b3cb-79f007881cc5 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:03:39.595414 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0cb3040-261a-4834-8898-06b07ea5fb7a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:03:39.596056 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0cb3040-261a-4834-8898-06b07ea5fb7a admin admin] REQUEST_METHOD: `PUT` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:03:39.596334 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0cb3040-261a-4834-8898-06b07ea5fb7a admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:03:39.596604 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0cb3040-261a-4834-8898-06b07ea5fb7a admin admin] PATH_INFO: `/v3/projects/eb436fc517954fccb46b1319a7cf44f7/users/68f168f3c223467d957939b0dc3d4b9f/roles/1fb81afcb9f245f3b989add94598a063` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:03:39.611537 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0cb3040-261a-4834-8898-06b07ea5fb7a admin admin] RBAC: Authorizing `identity:create_grant(project_id=eb436fc517954fccb46b1319a7cf44f7, user_id=68f168f3c223467d957939b0dc3d4b9f, role_id=1fb81afcb9f245f3b989add94598a063)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:03:39.614249 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0cb3040-261a-4834-8898-06b07ea5fb7a admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:03:39.746938 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1095/2180] 10.222.0.27 () {60 vars in 1483 bytes} [Tue Aug 30 02:03:39 2022] PUT /identity/v3/projects/eb436fc517954fccb46b1319a7cf44f7/users/68f168f3c223467d957939b0dc3d4b9f/roles/1fb81afcb9f245f3b989add94598a063 => generated 0 bytes in 167 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 02:03:39.757676 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c44dce7-248b-41e4-9f88-2bb6e3b4d97e None None] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:03:39.758678 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c44dce7-248b-41e4-9f88-2bb6e3b4d97e None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:03:39.758927 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c44dce7-248b-41e4-9f88-2bb6e3b4d97e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:03:39.776110 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=84,keystone:DELETE=13,keystone:INSERT=3 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:03:39.786687 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.auth.core [None req-7c44dce7-248b-41e4-9f88-2bb6e3b4d97e None None] MFA Rules not processed for user `68f168f3c223467d957939b0dc3d4b9f`. Rule list: `[]` (Enabled: `True`). {{(pid=81578) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:03:39.787887 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-7c44dce7-248b-41e4-9f88-2bb6e3b4d97e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:03:39.857300 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1086/2181] 10.222.0.27 () {58 vars in 958 bytes} [Tue Aug 30 02:03:39 2022] POST /identity/v3/auth/tokens => generated 3266 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 02:03:39.861363 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0cb3040-261a-4834-8898-06b07ea5fb7a admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:03:39.924847 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0cb3040-261a-4834-8898-06b07ea5fb7a admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:03:39.930484 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-887c5738-0da3-454d-9975-a59872cc1a6a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:03:39.931204 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-887c5738-0da3-454d-9975-a59872cc1a6a admin admin] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:03:39.931377 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-887c5738-0da3-454d-9975-a59872cc1a6a admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:03:39.931649 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-887c5738-0da3-454d-9975-a59872cc1a6a admin admin] PATH_INFO: `/v3/projects` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:03:39.932314 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-887c5738-0da3-454d-9975-a59872cc1a6a admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:03:39.934343 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-887c5738-0da3-454d-9975-a59872cc1a6a admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:03:40.058097 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1096/2182] 10.222.0.27 () {60 vars in 1153 bytes} [Tue Aug 30 02:03:39 2022] POST /identity/v3/projects => generated 387 bytes in 197 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 02:03:40.062079 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c44dce7-248b-41e4-9f88-2bb6e3b4d97e None None] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:03:40.078497 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c44dce7-248b-41e4-9f88-2bb6e3b4d97e None None] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:03:40.137648 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99c6a0df-bc70-423d-9843-98b5fc7a7821 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:03:40.138368 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99c6a0df-bc70-423d-9843-98b5fc7a7821 admin admin] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:03:40.138505 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99c6a0df-bc70-423d-9843-98b5fc7a7821 admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:03:40.138829 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99c6a0df-bc70-423d-9843-98b5fc7a7821 admin admin] PATH_INFO: `/v3/users` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:03:40.139438 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99c6a0df-bc70-423d-9843-98b5fc7a7821 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:03:40.141701 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99c6a0df-bc70-423d-9843-98b5fc7a7821 admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:03:40.318470 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1087/2183] 10.222.0.27 () {60 vars in 1144 bytes} [Tue Aug 30 02:03:40 2022] POST /identity/v3/users => generated 300 bytes in 257 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 02:03:40.322496 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-887c5738-0da3-454d-9975-a59872cc1a6a admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:03:40.332520 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-887c5738-0da3-454d-9975-a59872cc1a6a admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:03:40.339621 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42d7b80f-a6a3-44ab-9bca-c5a0e306d860 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:03:40.339621 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42d7b80f-a6a3-44ab-9bca-c5a0e306d860 admin admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:03:40.339621 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42d7b80f-a6a3-44ab-9bca-c5a0e306d860 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:03:40.339621 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42d7b80f-a6a3-44ab-9bca-c5a0e306d860 admin admin] PATH_INFO: `/v3/roles` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:03:40.340031 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42d7b80f-a6a3-44ab-9bca-c5a0e306d860 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:03:40.341002 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42d7b80f-a6a3-44ab-9bca-c5a0e306d860 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:03:40.353059 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1097/2184] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Aug 30 02:03:40 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 02:03:40.357379 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99c6a0df-bc70-423d-9843-98b5fc7a7821 admin admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:03:40.367451 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99c6a0df-bc70-423d-9843-98b5fc7a7821 admin admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:03:40.372356 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-519c2666-977b-4464-a341-ccb5aa87afb9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:03:40.373006 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-519c2666-977b-4464-a341-ccb5aa87afb9 admin admin] REQUEST_METHOD: `PUT` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:03:40.373281 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-519c2666-977b-4464-a341-ccb5aa87afb9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:03:40.373578 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-519c2666-977b-4464-a341-ccb5aa87afb9 admin admin] PATH_INFO: `/v3/projects/cb4bbceb425249ab8c5990c61624dd13/users/6dcd71082c3a4c3cbf0ee7bc2185d361/roles/ead5fac86a0747ea8823a82efcea8ed7` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:03:40.389288 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-519c2666-977b-4464-a341-ccb5aa87afb9 admin admin] RBAC: Authorizing `identity:create_grant(project_id=cb4bbceb425249ab8c5990c61624dd13, user_id=6dcd71082c3a4c3cbf0ee7bc2185d361, role_id=ead5fac86a0747ea8823a82efcea8ed7)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:03:40.392001 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-519c2666-977b-4464-a341-ccb5aa87afb9 admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:03:40.525531 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1088/2185] 10.222.0.27 () {60 vars in 1483 bytes} [Tue Aug 30 02:03:40 2022] PUT /identity/v3/projects/cb4bbceb425249ab8c5990c61624dd13/users/6dcd71082c3a4c3cbf0ee7bc2185d361/roles/ead5fac86a0747ea8823a82efcea8ed7 => generated 0 bytes in 169 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 02:03:40.529728 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42d7b80f-a6a3-44ab-9bca-c5a0e306d860 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:03:40.540100 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42d7b80f-a6a3-44ab-9bca-c5a0e306d860 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:03:40.606175 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d007fa2-733e-4b2a-9f40-17fbd7e63d90 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:03:40.606836 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d007fa2-733e-4b2a-9f40-17fbd7e63d90 admin admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:03:40.607115 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d007fa2-733e-4b2a-9f40-17fbd7e63d90 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:03:40.607386 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d007fa2-733e-4b2a-9f40-17fbd7e63d90 admin admin] PATH_INFO: `/v3/roles` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:03:40.608035 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d007fa2-733e-4b2a-9f40-17fbd7e63d90 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:03:40.609693 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d007fa2-733e-4b2a-9f40-17fbd7e63d90 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:03:40.619760 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1098/2186] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Aug 30 02:03:40 2022] GET /identity/v3/roles => generated 1420 bytes in 91 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 02:03:40.624070 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-519c2666-977b-4464-a341-ccb5aa87afb9 admin admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:03:40.634934 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-519c2666-977b-4464-a341-ccb5aa87afb9 admin admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:03:40.640277 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c285712-edfb-4f0a-825c-2c51e2d42937 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:03:40.640929 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c285712-edfb-4f0a-825c-2c51e2d42937 admin admin] REQUEST_METHOD: `PUT` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:03:40.641211 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c285712-edfb-4f0a-825c-2c51e2d42937 admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:03:40.641510 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c285712-edfb-4f0a-825c-2c51e2d42937 admin admin] PATH_INFO: `/v3/projects/cb4bbceb425249ab8c5990c61624dd13/users/6dcd71082c3a4c3cbf0ee7bc2185d361/roles/ead5fac86a0747ea8823a82efcea8ed7` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:03:40.643774 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c285712-edfb-4f0a-825c-2c51e2d42937 admin admin] RBAC: Authorizing `identity:create_grant(project_id=cb4bbceb425249ab8c5990c61624dd13, user_id=6dcd71082c3a4c3cbf0ee7bc2185d361, role_id=ead5fac86a0747ea8823a82efcea8ed7)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:03:40.646467 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c285712-edfb-4f0a-825c-2c51e2d42937 admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:03:40.657907 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1089/2187] 10.222.0.27 () {60 vars in 1483 bytes} [Tue Aug 30 02:03:40 2022] PUT /identity/v3/projects/cb4bbceb425249ab8c5990c61624dd13/users/6dcd71082c3a4c3cbf0ee7bc2185d361/roles/ead5fac86a0747ea8823a82efcea8ed7 => generated 0 bytes in 34 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 02:03:40.669910 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a6de25d-5007-4e18-9f47-df48b387e047 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:03:40.670201 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a6de25d-5007-4e18-9f47-df48b387e047 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:03:40.670478 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a6de25d-5007-4e18-9f47-df48b387e047 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:03:40.698600 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-1a6de25d-5007-4e18-9f47-df48b387e047 None None] MFA Rules not processed for user `6dcd71082c3a4c3cbf0ee7bc2185d361`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:03:40.699834 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-1a6de25d-5007-4e18-9f47-df48b387e047 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:03:40.771080 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1099/2188] 10.222.0.27 () {58 vars in 958 bytes} [Tue Aug 30 02:03:40 2022] POST /identity/v3/auth/tokens => generated 3331 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 02:03:40.782637 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c285712-edfb-4f0a-825c-2c51e2d42937 admin admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:03:40.784320 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-4c285712-edfb-4f0a-825c-2c51e2d42937 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:03:40.842975 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c285712-edfb-4f0a-825c-2c51e2d42937 admin admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:03:40.846732 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-568825c1-3616-4c49-8b5d-4d777593b660 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ab936b3767a8417aa80c8c8f1b84b667', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4562f3951fbe465280073e60a11e44e2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:03:40.847334 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-568825c1-3616-4c49-8b5d-4d777593b660 service cinder] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:03:40.847612 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-568825c1-3616-4c49-8b5d-4d777593b660 service cinder] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:03:40.847928 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-568825c1-3616-4c49-8b5d-4d777593b660 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:03:40.856965 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-568825c1-3616-4c49-8b5d-4d777593b660 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:03:40.858411 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-568825c1-3616-4c49-8b5d-4d777593b660 service cinder] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:03:40.867804 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1090/2189] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Aug 30 02:03:40 2022] GET /identity/v3/auth/tokens => generated 3331 bytes in 86 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 02:03:41.010534 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a6de25d-5007-4e18-9f47-df48b387e047 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:03:41.077484 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a6de25d-5007-4e18-9f47-df48b387e047 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:03:41.082422 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f3237b0-5ba0-479f-81cb-0631f6d0463e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:03:41.083223 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f3237b0-5ba0-479f-81cb-0631f6d0463e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:03:41.083486 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f3237b0-5ba0-479f-81cb-0631f6d0463e admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:03:41.083760 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f3237b0-5ba0-479f-81cb-0631f6d0463e admin admin] PATH_INFO: `/v3/users/68f168f3c223467d957939b0dc3d4b9f` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:03:41.085016 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f3237b0-5ba0-479f-81cb-0631f6d0463e admin admin] RBAC: Authorizing `identity:delete_user(user_id=68f168f3c223467d957939b0dc3d4b9f)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:03:41.087960 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f3237b0-5ba0-479f-81cb-0631f6d0463e admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:03:41.255332 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=81,keystone:DELETE=17,keystone:INSERT=2 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:03:41.455033 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-5f3237b0-5ba0-479f-81cb-0631f6d0463e admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '68f168f3c223467d957939b0dc3d4b9f'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:03:41.455693 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-5f3237b0-5ba0-479f-81cb-0631f6d0463e admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '68f168f3c223467d957939b0dc3d4b9f'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:03:41.456348 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-5f3237b0-5ba0-479f-81cb-0631f6d0463e admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '68f168f3c223467d957939b0dc3d4b9f'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:03:41.469080 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-5f3237b0-5ba0-479f-81cb-0631f6d0463e admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '68f168f3c223467d957939b0dc3d4b9f'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:03:41.475739 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-5f3237b0-5ba0-479f-81cb-0631f6d0463e admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '68f168f3c223467d957939b0dc3d4b9f'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:03:41.652094 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1100/2190] 10.222.0.27 () {58 vars in 1224 bytes} [Tue Aug 30 02:03:41 2022] DELETE /identity/v3/users/68f168f3c223467d957939b0dc3d4b9f => generated 0 bytes in 640 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 02:03:41.654425 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ea3d6f5-efca-4513-9828-bec608ddad1b None None] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:03:41.654893 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ea3d6f5-efca-4513-9828-bec608ddad1b None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:03:41.655226 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ea3d6f5-efca-4513-9828-bec608ddad1b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:03:41.675539 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.auth.core [None req-5ea3d6f5-efca-4513-9828-bec608ddad1b None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81578) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:03:41.676983 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-5ea3d6f5-efca-4513-9828-bec608ddad1b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:03:41.737380 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1091/2191] 10.222.0.27 () {58 vars in 958 bytes} [Tue Aug 30 02:03:41 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 02:03:41.747846 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f3237b0-5ba0-479f-81cb-0631f6d0463e admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:03:41.749528 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-5f3237b0-5ba0-479f-81cb-0631f6d0463e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:03:41.811334 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f3237b0-5ba0-479f-81cb-0631f6d0463e admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:03:41.814872 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3de1c8e-c8f4-4c84-b8ab-494ccd764d35 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f3ea1580c56f4814a92bbb2acf542905', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4562f3951fbe465280073e60a11e44e2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:03:41.815602 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3de1c8e-c8f4-4c84-b8ab-494ccd764d35 service neutron] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:03:41.815882 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3de1c8e-c8f4-4c84-b8ab-494ccd764d35 service neutron] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:03:41.816159 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3de1c8e-c8f4-4c84-b8ab-494ccd764d35 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:03:41.825628 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3de1c8e-c8f4-4c84-b8ab-494ccd764d35 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:03:41.827060 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3de1c8e-c8f4-4c84-b8ab-494ccd764d35 service neutron] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:03:41.836851 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1101/2192] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Aug 30 02:03:41 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 90 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 02:03:42.020855 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:03:42.429628 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ea3d6f5-efca-4513-9828-bec608ddad1b None None] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:03:42.432528 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-5ea3d6f5-efca-4513-9828-bec608ddad1b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:03:42.451254 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ea3d6f5-efca-4513-9828-bec608ddad1b None None] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:03:42.455928 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbffc569-380d-433e-bfa8-3327a628d97e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:03:42.456560 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbffc569-380d-433e-bfa8-3327a628d97e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:03:42.456836 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbffc569-380d-433e-bfa8-3327a628d97e admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:03:42.457110 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbffc569-380d-433e-bfa8-3327a628d97e admin admin] PATH_INFO: `/v3/projects/eb436fc517954fccb46b1319a7cf44f7` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:03:42.465888 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbffc569-380d-433e-bfa8-3327a628d97e admin admin] RBAC: Authorizing `identity:delete_project(project_id=eb436fc517954fccb46b1319a7cf44f7)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:03:42.468074 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbffc569-380d-433e-bfa8-3327a628d97e admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:03:42.611709 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-fbffc569-380d-433e-bfa8-3327a628d97e admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'eb436fc517954fccb46b1319a7cf44f7', 'request_id': 'req-fbffc569-380d-433e-bfa8-3327a628d97e'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:03:42.621260 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-fbffc569-380d-433e-bfa8-3327a628d97e admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'eb436fc517954fccb46b1319a7cf44f7', 'request_id': 'req-fbffc569-380d-433e-bfa8-3327a628d97e'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:03:42.634250 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-fbffc569-380d-433e-bfa8-3327a628d97e admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'eb436fc517954fccb46b1319a7cf44f7', 'request_id': 'req-fbffc569-380d-433e-bfa8-3327a628d97e'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:03:42.757483 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-fbffc569-380d-433e-bfa8-3327a628d97e admin admin] The token cache is being invalidate because project eb436fc517954fccb46b1319a7cf44f7 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81578) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 30 02:03:42.758789 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-fbffc569-380d-433e-bfa8-3327a628d97e admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:03:42.759455 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-fbffc569-380d-433e-bfa8-3327a628d97e admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:03:42.760936 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1092/2193] 10.222.0.27 () {58 vars in 1233 bytes} [Tue Aug 30 02:03:42 2022] DELETE /identity/v3/projects/eb436fc517954fccb46b1319a7cf44f7 => generated 0 bytes in 332 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 02:03:42.764961 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3de1c8e-c8f4-4c84-b8ab-494ccd764d35 service neutron] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:03:42.774646 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3de1c8e-c8f4-4c84-b8ab-494ccd764d35 service neutron] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:03:42.830623 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5419e38-81fb-4f92-80da-07e1f1950636 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:03:42.831321 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5419e38-81fb-4f92-80da-07e1f1950636 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:03:42.831637 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5419e38-81fb-4f92-80da-07e1f1950636 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:03:42.831905 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5419e38-81fb-4f92-80da-07e1f1950636 admin admin] PATH_INFO: `/v3/users/6dcd71082c3a4c3cbf0ee7bc2185d361` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:03:42.833015 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5419e38-81fb-4f92-80da-07e1f1950636 admin admin] RBAC: Authorizing `identity:delete_user(user_id=6dcd71082c3a4c3cbf0ee7bc2185d361)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:03:42.835231 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5419e38-81fb-4f92-80da-07e1f1950636 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:03:42.855014 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:03:42.906502 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:03:43.091764 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-a5419e38-81fb-4f92-80da-07e1f1950636 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '6dcd71082c3a4c3cbf0ee7bc2185d361'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:03:43.092413 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-a5419e38-81fb-4f92-80da-07e1f1950636 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '6dcd71082c3a4c3cbf0ee7bc2185d361'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:03:43.093060 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-a5419e38-81fb-4f92-80da-07e1f1950636 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '6dcd71082c3a4c3cbf0ee7bc2185d361'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:03:43.105391 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-a5419e38-81fb-4f92-80da-07e1f1950636 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '6dcd71082c3a4c3cbf0ee7bc2185d361'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:03:43.112111 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-a5419e38-81fb-4f92-80da-07e1f1950636 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '6dcd71082c3a4c3cbf0ee7bc2185d361'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:03:43.260507 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1102/2194] 10.222.0.27 () {58 vars in 1224 bytes} [Tue Aug 30 02:03:42 2022] DELETE /identity/v3/users/6dcd71082c3a4c3cbf0ee7bc2185d361 => generated 0 bytes in 496 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 02:03:43.821775 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbffc569-380d-433e-bfa8-3327a628d97e admin admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:03:43.823327 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-fbffc569-380d-433e-bfa8-3327a628d97e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:03:43.839079 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbffc569-380d-433e-bfa8-3327a628d97e admin admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:03:43.905915 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ae7e41b-23c7-4211-9e57-50cd7f5e3d9d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:03:43.906560 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ae7e41b-23c7-4211-9e57-50cd7f5e3d9d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:03:43.906842 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ae7e41b-23c7-4211-9e57-50cd7f5e3d9d admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:03:43.907114 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ae7e41b-23c7-4211-9e57-50cd7f5e3d9d admin admin] PATH_INFO: `/v3/projects/cb4bbceb425249ab8c5990c61624dd13` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:03:43.908375 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ae7e41b-23c7-4211-9e57-50cd7f5e3d9d admin admin] RBAC: Authorizing `identity:delete_project(project_id=cb4bbceb425249ab8c5990c61624dd13)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:03:43.910307 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ae7e41b-23c7-4211-9e57-50cd7f5e3d9d admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:03:44.091646 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-5ae7e41b-23c7-4211-9e57-50cd7f5e3d9d admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'cb4bbceb425249ab8c5990c61624dd13', 'request_id': 'req-5ae7e41b-23c7-4211-9e57-50cd7f5e3d9d'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:03:44.097536 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-5ae7e41b-23c7-4211-9e57-50cd7f5e3d9d admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'cb4bbceb425249ab8c5990c61624dd13', 'request_id': 'req-5ae7e41b-23c7-4211-9e57-50cd7f5e3d9d'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:03:44.104102 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-5ae7e41b-23c7-4211-9e57-50cd7f5e3d9d admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'cb4bbceb425249ab8c5990c61624dd13', 'request_id': 'req-5ae7e41b-23c7-4211-9e57-50cd7f5e3d9d'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:03:44.225941 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-5ae7e41b-23c7-4211-9e57-50cd7f5e3d9d admin admin] The token cache is being invalidate because project cb4bbceb425249ab8c5990c61624dd13 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81578) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 30 02:03:44.226225 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-5ae7e41b-23c7-4211-9e57-50cd7f5e3d9d admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:03:44.226971 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-5ae7e41b-23c7-4211-9e57-50cd7f5e3d9d admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:03:44.228451 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1093/2195] 10.222.0.27 () {58 vars in 1233 bytes} [Tue Aug 30 02:03:43 2022] DELETE /identity/v3/projects/cb4bbceb425249ab8c5990c61624dd13 => generated 0 bytes in 408 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 02:03:44.233579 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11cf3afd-bb7f-4e80-aca9-0df288614a12 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:03:44.234065 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11cf3afd-bb7f-4e80-aca9-0df288614a12 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:03:44.234158 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11cf3afd-bb7f-4e80-aca9-0df288614a12 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:03:44.254180 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-11cf3afd-bb7f-4e80-aca9-0df288614a12 None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:03:44.255393 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-11cf3afd-bb7f-4e80-aca9-0df288614a12 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:03:44.314021 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1103/2196] 10.222.0.27 () {58 vars in 958 bytes} [Tue Aug 30 02:03:44 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 81 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 02:03:44.322954 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79740267-1aa0-4573-a45a-ee296b3983a9 None None] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:03:44.323172 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79740267-1aa0-4573-a45a-ee296b3983a9 None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:03:44.323441 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79740267-1aa0-4573-a45a-ee296b3983a9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:03:44.344507 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.auth.core [None req-79740267-1aa0-4573-a45a-ee296b3983a9 None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81578) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:03:44.346159 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-79740267-1aa0-4573-a45a-ee296b3983a9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:03:44.354399 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1094/2197] 10.222.0.27 () {58 vars in 958 bytes} [Tue Aug 30 02:03:44 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 02:03:44.358788 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11cf3afd-bb7f-4e80-aca9-0df288614a12 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:03:44.368943 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11cf3afd-bb7f-4e80-aca9-0df288614a12 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:03:44.373621 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01db633e-86ec-4196-b67a-807b8955f72e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:03:44.374250 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01db633e-86ec-4196-b67a-807b8955f72e admin admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:03:44.374630 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01db633e-86ec-4196-b67a-807b8955f72e admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:03:44.374948 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01db633e-86ec-4196-b67a-807b8955f72e admin admin] PATH_INFO: `/v3/domains` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:03:44.375608 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01db633e-86ec-4196-b67a-807b8955f72e admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 02:03:44.378294 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01db633e-86ec-4196-b67a-807b8955f72e admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:03:44.378294 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01db633e-86ec-4196-b67a-807b8955f72e admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:03:44.390349 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1104/2198] 10.222.0.27 () {58 vars in 1153 bytes} [Tue Aug 30 02:03:44 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 02:03:44.394717 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79740267-1aa0-4573-a45a-ee296b3983a9 None None] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:03:44.404414 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79740267-1aa0-4573-a45a-ee296b3983a9 None None] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:03:44.409267 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5023ba8e-b846-4366-94e5-3d9384aa8392 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:03:44.409859 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5023ba8e-b846-4366-94e5-3d9384aa8392 admin admin] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:03:44.410157 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5023ba8e-b846-4366-94e5-3d9384aa8392 admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:03:44.410430 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5023ba8e-b846-4366-94e5-3d9384aa8392 admin admin] PATH_INFO: `/v3/projects` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:03:44.411062 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5023ba8e-b846-4366-94e5-3d9384aa8392 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:03:44.413090 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5023ba8e-b846-4366-94e5-3d9384aa8392 admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:03:44.536050 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1095/2199] 10.222.0.27 () {60 vars in 1153 bytes} [Tue Aug 30 02:03:44 2022] POST /identity/v3/projects => generated 385 bytes in 142 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 30 02:03:44.540885 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01db633e-86ec-4196-b67a-807b8955f72e admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:03:44.550811 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01db633e-86ec-4196-b67a-807b8955f72e admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:03:44.607787 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96080335-41b5-42f2-9b95-0537af8286c5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:03:44.608347 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96080335-41b5-42f2-9b95-0537af8286c5 admin admin] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:03:44.608621 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96080335-41b5-42f2-9b95-0537af8286c5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:03:44.608892 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96080335-41b5-42f2-9b95-0537af8286c5 admin admin] PATH_INFO: `/v3/users` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:03:44.609558 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96080335-41b5-42f2-9b95-0537af8286c5 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:03:44.611494 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96080335-41b5-42f2-9b95-0537af8286c5 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:03:44.710041 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1105/2200] 10.222.0.27 () {60 vars in 1144 bytes} [Tue Aug 30 02:03:44 2022] POST /identity/v3/users => generated 293 bytes in 167 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 02:03:44.711392 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5023ba8e-b846-4366-94e5-3d9384aa8392 admin admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:03:44.722755 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5023ba8e-b846-4366-94e5-3d9384aa8392 admin admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:03:44.728320 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe75b91e-5e52-4bc5-9c2b-7af4d63759cc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:03:44.728552 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe75b91e-5e52-4bc5-9c2b-7af4d63759cc admin admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:03:44.728892 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe75b91e-5e52-4bc5-9c2b-7af4d63759cc admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:03:44.729167 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe75b91e-5e52-4bc5-9c2b-7af4d63759cc admin admin] PATH_INFO: `/v3/roles` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:03:44.729920 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe75b91e-5e52-4bc5-9c2b-7af4d63759cc admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:03:44.731575 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe75b91e-5e52-4bc5-9c2b-7af4d63759cc admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:03:44.742000 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1096/2201] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Aug 30 02:03:44 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 02:03:44.746659 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96080335-41b5-42f2-9b95-0537af8286c5 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:03:44.756878 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96080335-41b5-42f2-9b95-0537af8286c5 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:03:44.761853 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b574e216-e0a8-4142-a8df-6f982d2afb86 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:03:44.762459 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b574e216-e0a8-4142-a8df-6f982d2afb86 admin admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:03:44.762741 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b574e216-e0a8-4142-a8df-6f982d2afb86 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:03:44.763016 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b574e216-e0a8-4142-a8df-6f982d2afb86 admin admin] PATH_INFO: `/v3/roles` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:03:44.763660 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b574e216-e0a8-4142-a8df-6f982d2afb86 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:03:44.765263 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b574e216-e0a8-4142-a8df-6f982d2afb86 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:03:44.776308 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1106/2202] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Aug 30 02:03:44 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 30 02:03:44.780995 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe75b91e-5e52-4bc5-9c2b-7af4d63759cc admin admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:03:44.792362 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe75b91e-5e52-4bc5-9c2b-7af4d63759cc admin admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:03:44.797517 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86562f43-463e-4240-8420-e8f0760ec9fc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:03:44.798272 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86562f43-463e-4240-8420-e8f0760ec9fc admin admin] REQUEST_METHOD: `PUT` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:03:44.798625 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86562f43-463e-4240-8420-e8f0760ec9fc admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:03:44.798970 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86562f43-463e-4240-8420-e8f0760ec9fc admin admin] PATH_INFO: `/v3/projects/f29531622fd74dee8e50fa700bf63f04/users/a579fb8e4edd4faea19b9675c22ac4ab/roles/1fb81afcb9f245f3b989add94598a063` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:03:44.814714 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86562f43-463e-4240-8420-e8f0760ec9fc admin admin] RBAC: Authorizing `identity:create_grant(project_id=f29531622fd74dee8e50fa700bf63f04, user_id=a579fb8e4edd4faea19b9675c22ac4ab, role_id=1fb81afcb9f245f3b989add94598a063)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:03:44.817359 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86562f43-463e-4240-8420-e8f0760ec9fc admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:03:44.918607 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1097/2203] 10.222.0.27 () {60 vars in 1483 bytes} [Tue Aug 30 02:03:44 2022] PUT /identity/v3/projects/f29531622fd74dee8e50fa700bf63f04/users/a579fb8e4edd4faea19b9675c22ac4ab/roles/1fb81afcb9f245f3b989add94598a063 => generated 0 bytes in 138 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 30 02:03:44.928900 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c05057e-25b6-4e55-9e36-1018be813211 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:03:44.929198 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c05057e-25b6-4e55-9e36-1018be813211 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:03:44.930003 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c05057e-25b6-4e55-9e36-1018be813211 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:03:44.958664 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-1c05057e-25b6-4e55-9e36-1018be813211 None None] MFA Rules not processed for user `a579fb8e4edd4faea19b9675c22ac4ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:03:44.959980 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-1c05057e-25b6-4e55-9e36-1018be813211 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:03:45.037001 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1107/2204] 10.222.0.27 () {58 vars in 958 bytes} [Tue Aug 30 02:03:44 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 02:03:45.043099 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86562f43-463e-4240-8420-e8f0760ec9fc admin admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:03:45.048601 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:03:45.117894 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86562f43-463e-4240-8420-e8f0760ec9fc admin admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:03:45.123996 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1a5f98d-74df-4b47-87b2-bed97af1ee5c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:03:45.124958 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1a5f98d-74df-4b47-87b2-bed97af1ee5c admin admin] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:03:45.125319 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1a5f98d-74df-4b47-87b2-bed97af1ee5c admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:03:45.125737 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1a5f98d-74df-4b47-87b2-bed97af1ee5c admin admin] PATH_INFO: `/v3/projects` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:03:45.126401 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1a5f98d-74df-4b47-87b2-bed97af1ee5c admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:03:45.128577 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1a5f98d-74df-4b47-87b2-bed97af1ee5c admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:03:45.341753 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1098/2205] 10.222.0.27 () {60 vars in 1153 bytes} [Tue Aug 30 02:03:45 2022] POST /identity/v3/projects => generated 385 bytes in 300 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 02:03:45.345592 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c05057e-25b6-4e55-9e36-1018be813211 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:03:45.357220 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c05057e-25b6-4e55-9e36-1018be813211 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:03:45.416102 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af4e1116-f4fa-4b06-86f0-58aabfa44271 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:03:45.416727 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af4e1116-f4fa-4b06-86f0-58aabfa44271 admin admin] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:03:45.417008 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af4e1116-f4fa-4b06-86f0-58aabfa44271 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:03:45.417280 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af4e1116-f4fa-4b06-86f0-58aabfa44271 admin admin] PATH_INFO: `/v3/users` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:03:45.417958 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af4e1116-f4fa-4b06-86f0-58aabfa44271 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:03:45.419952 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af4e1116-f4fa-4b06-86f0-58aabfa44271 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:03:45.438091 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:03:45.541975 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1108/2206] 10.222.0.27 () {60 vars in 1144 bytes} [Tue Aug 30 02:03:45 2022] POST /identity/v3/users => generated 299 bytes in 197 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 02:03:45.556969 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1a5f98d-74df-4b47-87b2-bed97af1ee5c admin admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:03:45.567429 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1a5f98d-74df-4b47-87b2-bed97af1ee5c admin admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:03:45.572161 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-685f9981-89e0-4f0e-91d0-a78f1ff97fae admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:03:45.572747 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-685f9981-89e0-4f0e-91d0-a78f1ff97fae admin admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:03:45.573021 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-685f9981-89e0-4f0e-91d0-a78f1ff97fae admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:03:45.573289 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-685f9981-89e0-4f0e-91d0-a78f1ff97fae admin admin] PATH_INFO: `/v3/roles` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:03:45.573961 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-685f9981-89e0-4f0e-91d0-a78f1ff97fae admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:03:45.575528 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-685f9981-89e0-4f0e-91d0-a78f1ff97fae admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:03:45.586590 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1099/2207] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Aug 30 02:03:45 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 02:03:45.591418 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af4e1116-f4fa-4b06-86f0-58aabfa44271 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:03:45.602044 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af4e1116-f4fa-4b06-86f0-58aabfa44271 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:03:45.607054 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a10cc57a-e50d-4caf-b83f-94856d4e3832 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:03:45.607733 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a10cc57a-e50d-4caf-b83f-94856d4e3832 admin admin] REQUEST_METHOD: `PUT` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:03:45.608014 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a10cc57a-e50d-4caf-b83f-94856d4e3832 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:03:45.608290 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a10cc57a-e50d-4caf-b83f-94856d4e3832 admin admin] PATH_INFO: `/v3/projects/b6275eb14bac45e39bc26836dd7812ba/users/57e8680db0db4557ad5949849c10e681/roles/ead5fac86a0747ea8823a82efcea8ed7` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:03:45.623556 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a10cc57a-e50d-4caf-b83f-94856d4e3832 admin admin] RBAC: Authorizing `identity:create_grant(project_id=b6275eb14bac45e39bc26836dd7812ba, user_id=57e8680db0db4557ad5949849c10e681, role_id=ead5fac86a0747ea8823a82efcea8ed7)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:03:45.626254 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a10cc57a-e50d-4caf-b83f-94856d4e3832 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:03:45.775435 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1109/2208] 10.222.0.27 () {60 vars in 1483 bytes} [Tue Aug 30 02:03:45 2022] PUT /identity/v3/projects/b6275eb14bac45e39bc26836dd7812ba/users/57e8680db0db4557ad5949849c10e681/roles/ead5fac86a0747ea8823a82efcea8ed7 => generated 0 bytes in 185 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 02:03:45.779406 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-685f9981-89e0-4f0e-91d0-a78f1ff97fae admin admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:03:45.789900 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-685f9981-89e0-4f0e-91d0-a78f1ff97fae admin admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:03:45.859371 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af66c292-c96d-4537-863e-f53c506e9dfd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:03:45.860051 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af66c292-c96d-4537-863e-f53c506e9dfd admin admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:03:45.860395 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af66c292-c96d-4537-863e-f53c506e9dfd admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:03:45.860676 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af66c292-c96d-4537-863e-f53c506e9dfd admin admin] PATH_INFO: `/v3/roles` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:03:45.861351 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af66c292-c96d-4537-863e-f53c506e9dfd admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:03:45.862963 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af66c292-c96d-4537-863e-f53c506e9dfd admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:03:45.873342 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1100/2209] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Aug 30 02:03:45 2022] GET /identity/v3/roles => generated 1420 bytes in 94 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 02:03:45.877562 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a10cc57a-e50d-4caf-b83f-94856d4e3832 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:03:45.888695 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a10cc57a-e50d-4caf-b83f-94856d4e3832 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:03:45.893363 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4e5068b-1455-46b3-9a84-31025b35ea2d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:03:45.894059 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4e5068b-1455-46b3-9a84-31025b35ea2d admin admin] REQUEST_METHOD: `PUT` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:03:45.894340 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4e5068b-1455-46b3-9a84-31025b35ea2d admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:03:45.894615 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4e5068b-1455-46b3-9a84-31025b35ea2d admin admin] PATH_INFO: `/v3/projects/b6275eb14bac45e39bc26836dd7812ba/users/57e8680db0db4557ad5949849c10e681/roles/ead5fac86a0747ea8823a82efcea8ed7` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:03:45.896533 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4e5068b-1455-46b3-9a84-31025b35ea2d admin admin] RBAC: Authorizing `identity:create_grant(project_id=b6275eb14bac45e39bc26836dd7812ba, user_id=57e8680db0db4557ad5949849c10e681, role_id=ead5fac86a0747ea8823a82efcea8ed7)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:03:45.899611 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4e5068b-1455-46b3-9a84-31025b35ea2d admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:03:45.912731 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1110/2210] 10.222.0.27 () {60 vars in 1483 bytes} [Tue Aug 30 02:03:45 2022] PUT /identity/v3/projects/b6275eb14bac45e39bc26836dd7812ba/users/57e8680db0db4557ad5949849c10e681/roles/ead5fac86a0747ea8823a82efcea8ed7 => generated 0 bytes in 36 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 30 02:03:45.924184 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7957b8f9-3037-43f2-8a54-5de6ef7f1ee5 None None] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:03:45.924670 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7957b8f9-3037-43f2-8a54-5de6ef7f1ee5 None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:03:45.924869 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7957b8f9-3037-43f2-8a54-5de6ef7f1ee5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:03:45.954551 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.auth.core [None req-7957b8f9-3037-43f2-8a54-5de6ef7f1ee5 None None] MFA Rules not processed for user `57e8680db0db4557ad5949849c10e681`. Rule list: `[]` (Enabled: `True`). {{(pid=81578) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:03:45.956171 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-7957b8f9-3037-43f2-8a54-5de6ef7f1ee5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:03:46.029057 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1101/2211] 10.222.0.27 () {58 vars in 958 bytes} [Tue Aug 30 02:03:45 2022] POST /identity/v3/auth/tokens => generated 3329 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 02:03:46.074979 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4e5068b-1455-46b3-9a84-31025b35ea2d admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:03:46.077093 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-b4e5068b-1455-46b3-9a84-31025b35ea2d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:03:46.137139 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4e5068b-1455-46b3-9a84-31025b35ea2d admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:03:46.141003 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1853802a-308f-4106-ab41-71886ac4619a service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ab936b3767a8417aa80c8c8f1b84b667', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4562f3951fbe465280073e60a11e44e2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:03:46.141835 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1853802a-308f-4106-ab41-71886ac4619a service cinder] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:03:46.142118 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1853802a-308f-4106-ab41-71886ac4619a service cinder] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:03:46.142384 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1853802a-308f-4106-ab41-71886ac4619a service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:03:46.205593 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1853802a-308f-4106-ab41-71886ac4619a service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:03:46.207038 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1853802a-308f-4106-ab41-71886ac4619a service cinder] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:03:46.216118 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1111/2212] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Aug 30 02:03:46 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 142 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 02:03:46.991793 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:03:47.581430 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:03:47.838193 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=88,keystone:INSERT=5,keystone:DELETE=7 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:03:48.346166 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:03:48.734667 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=85,keystone:DELETE=13,keystone:INSERT=2 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:03:48.986420 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=81,keystone:DELETE=17,keystone:INSERT=2 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:03:49.689236 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:03:50.100955 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:03:50.884010 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:03:51.031814 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=94,keystone:INSERT=5,keystone:DELETE=1 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:03:51.884186 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:03:52.357749 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=87,keystone:DELETE=12,keystone:INSERT=1 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:03:52.557705 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:03:52.821224 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=94,keystone:DELETE=6 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:03:52.865916 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=85,keystone:DELETE=11,keystone:INSERT=4 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:03:52.940314 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=92,keystone:DELETE=4,keystone:INSERT=4 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:03:53.140306 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:03:53.437728 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:03:53.523339 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:03:53.614815 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:03:53.643471 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=81,keystone:DELETE=17,keystone:INSERT=2 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:03:53.780288 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:03:53.808825 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:03:53.900349 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:INSERT=3,keystone:SELECT=92,keystone:DELETE=5 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:03:54.248846 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:03:54.454519 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:DELETE=9,keystone:SELECT=88,keystone:INSERT=3 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:03:54.494714 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:03:54.747296 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:03:54.820176 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=88,keystone:DELETE=10,keystone:INSERT=2 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:03:55.169409 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:03:55.226582 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:INSERT=4,keystone:SELECT=96 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:03:55.523492 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:03:55.660585 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:03:56.020431 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=88,keystone:INSERT=2,keystone:DELETE=10 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:03:56.340398 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:03:56.696700 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=85,keystone:INSERT=5,keystone:DELETE=10 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:03:56.992281 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:03:57.363087 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=90,keystone:DELETE=9,keystone:INSERT=1 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:03:57.465596 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:03:57.588588 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7957b8f9-3037-43f2-8a54-5de6ef7f1ee5 None None] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:03:57.590081 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-7957b8f9-3037-43f2-8a54-5de6ef7f1ee5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:03:57.603220 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7957b8f9-3037-43f2-8a54-5de6ef7f1ee5 None None] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:03:57.606839 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47e0d72b-eb27-4603-8392-b656868e75fe service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ab936b3767a8417aa80c8c8f1b84b667', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4562f3951fbe465280073e60a11e44e2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:03:57.607429 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47e0d72b-eb27-4603-8392-b656868e75fe service cinder] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:03:57.607703 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47e0d72b-eb27-4603-8392-b656868e75fe service cinder] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:03:57.607972 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47e0d72b-eb27-4603-8392-b656868e75fe service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:03:57.617288 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47e0d72b-eb27-4603-8392-b656868e75fe service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:03:57.619034 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47e0d72b-eb27-4603-8392-b656868e75fe service cinder] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:03:57.629886 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1102/2213] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Aug 30 02:03:57 2022] GET /identity/v3/auth/tokens => generated 3329 bytes in 42 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 02:03:57.940819 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1853802a-308f-4106-ab41-71886ac4619a service cinder] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:03:57.944207 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-1853802a-308f-4106-ab41-71886ac4619a service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:03:58.047562 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1853802a-308f-4106-ab41-71886ac4619a service cinder] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:03:58.057721 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed329292-23b3-493b-a407-8b5e5ecfaf05 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:03:58.067441 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed329292-23b3-493b-a407-8b5e5ecfaf05 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:03:58.067441 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed329292-23b3-493b-a407-8b5e5ecfaf05 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:03:58.067441 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed329292-23b3-493b-a407-8b5e5ecfaf05 admin admin] PATH_INFO: `/v3/users/358c3080807d412485ad516b644f5b47` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:03:58.067441 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed329292-23b3-493b-a407-8b5e5ecfaf05 admin admin] RBAC: Authorizing `identity:delete_user(user_id=358c3080807d412485ad516b644f5b47)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:03:58.067441 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed329292-23b3-493b-a407-8b5e5ecfaf05 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:03:58.360509 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-ed329292-23b3-493b-a407-8b5e5ecfaf05 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '358c3080807d412485ad516b644f5b47'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:03:58.361168 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-ed329292-23b3-493b-a407-8b5e5ecfaf05 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '358c3080807d412485ad516b644f5b47'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:03:58.361776 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-ed329292-23b3-493b-a407-8b5e5ecfaf05 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '358c3080807d412485ad516b644f5b47'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:03:58.373908 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-ed329292-23b3-493b-a407-8b5e5ecfaf05 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '358c3080807d412485ad516b644f5b47'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:03:58.380583 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-ed329292-23b3-493b-a407-8b5e5ecfaf05 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '358c3080807d412485ad516b644f5b47'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:03:58.503149 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1112/2214] 10.222.0.27 () {58 vars in 1224 bytes} [Tue Aug 30 02:03:57 2022] DELETE /identity/v3/users/358c3080807d412485ad516b644f5b47 => generated 0 bytes in 562 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 30 02:03:58.506857 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af98ac1d-32f4-4e39-83f6-69ae693309db None None] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:03:58.507450 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af98ac1d-32f4-4e39-83f6-69ae693309db None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:03:58.507450 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af98ac1d-32f4-4e39-83f6-69ae693309db None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:03:58.529253 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=91,keystone:DELETE=8,keystone:INSERT=1 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:03:58.531816 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.auth.core [None req-af98ac1d-32f4-4e39-83f6-69ae693309db None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81578) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:03:58.533159 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-af98ac1d-32f4-4e39-83f6-69ae693309db None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:03:58.616421 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1103/2215] 10.222.0.27 () {58 vars in 958 bytes} [Tue Aug 30 02:03:58 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 111 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 02:03:58.626472 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed329292-23b3-493b-a407-8b5e5ecfaf05 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:03:58.628676 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-ed329292-23b3-493b-a407-8b5e5ecfaf05 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:03:58.666663 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:03:58.691493 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed329292-23b3-493b-a407-8b5e5ecfaf05 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:03:58.695898 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd670dff-eacb-4745-b753-2ea1ea91e3f0 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f3ea1580c56f4814a92bbb2acf542905', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4562f3951fbe465280073e60a11e44e2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:03:58.696519 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd670dff-eacb-4745-b753-2ea1ea91e3f0 service neutron] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:03:58.696850 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd670dff-eacb-4745-b753-2ea1ea91e3f0 service neutron] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:03:58.697125 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd670dff-eacb-4745-b753-2ea1ea91e3f0 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:03:58.707105 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd670dff-eacb-4745-b753-2ea1ea91e3f0 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:03:58.708558 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd670dff-eacb-4745-b753-2ea1ea91e3f0 service neutron] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:03:58.719746 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1113/2216] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Aug 30 02:03:58 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 94 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 02:03:59.229277 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af98ac1d-32f4-4e39-83f6-69ae693309db None None] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:03:59.230826 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-af98ac1d-32f4-4e39-83f6-69ae693309db None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:03:59.245428 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af98ac1d-32f4-4e39-83f6-69ae693309db None None] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:03:59.250272 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a888961a-cb3d-4313-8e19-afe46a26150f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:03:59.250935 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a888961a-cb3d-4313-8e19-afe46a26150f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:03:59.251215 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a888961a-cb3d-4313-8e19-afe46a26150f admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:03:59.251485 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a888961a-cb3d-4313-8e19-afe46a26150f admin admin] PATH_INFO: `/v3/projects/df11f3896394474499599fda2fe2289b` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:03:59.252880 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a888961a-cb3d-4313-8e19-afe46a26150f admin admin] RBAC: Authorizing `identity:delete_project(project_id=df11f3896394474499599fda2fe2289b)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:03:59.254854 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a888961a-cb3d-4313-8e19-afe46a26150f admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:03:59.524308 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-a888961a-cb3d-4313-8e19-afe46a26150f admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'df11f3896394474499599fda2fe2289b', 'request_id': 'req-a888961a-cb3d-4313-8e19-afe46a26150f'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:03:59.529913 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-a888961a-cb3d-4313-8e19-afe46a26150f admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'df11f3896394474499599fda2fe2289b', 'request_id': 'req-a888961a-cb3d-4313-8e19-afe46a26150f'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:03:59.535985 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-a888961a-cb3d-4313-8e19-afe46a26150f admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'df11f3896394474499599fda2fe2289b', 'request_id': 'req-a888961a-cb3d-4313-8e19-afe46a26150f'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:03:59.640870 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd670dff-eacb-4745-b753-2ea1ea91e3f0 service neutron] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:03:59.642443 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-cd670dff-eacb-4745-b753-2ea1ea91e3f0 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:03:59.650871 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=86,keystone:DELETE=13,keystone:INSERT=1 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:03:59.658251 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd670dff-eacb-4745-b753-2ea1ea91e3f0 service neutron] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:03:59.660254 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-a888961a-cb3d-4313-8e19-afe46a26150f admin admin] The token cache is being invalidate because project df11f3896394474499599fda2fe2289b was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81578) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 30 02:03:59.660762 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-a888961a-cb3d-4313-8e19-afe46a26150f admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:03:59.661603 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-a888961a-cb3d-4313-8e19-afe46a26150f admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:03:59.663547 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1104/2217] 10.222.0.27 () {58 vars in 1233 bytes} [Tue Aug 30 02:03:59 2022] DELETE /identity/v3/projects/df11f3896394474499599fda2fe2289b => generated 0 bytes in 435 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 02:03:59.668227 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02c042fa-87cb-4367-a2bf-51f337e7abdb None None] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:03:59.668727 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02c042fa-87cb-4367-a2bf-51f337e7abdb None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:03:59.669068 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02c042fa-87cb-4367-a2bf-51f337e7abdb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:03:59.689141 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.auth.core [None req-02c042fa-87cb-4367-a2bf-51f337e7abdb None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81578) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:03:59.690453 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-02c042fa-87cb-4367-a2bf-51f337e7abdb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:03:59.727800 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c087fad8-17f1-43c0-8faa-3f6e24d37ec0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:03:59.728902 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c087fad8-17f1-43c0-8faa-3f6e24d37ec0 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:03:59.729293 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c087fad8-17f1-43c0-8faa-3f6e24d37ec0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:03:59.729631 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c087fad8-17f1-43c0-8faa-3f6e24d37ec0 admin admin] PATH_INFO: `/v3/users/6f1e736fab4f47ac868d1bed7400c571` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:03:59.731410 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=81,keystone:DELETE=17,keystone:INSERT=2 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:03:59.735787 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c087fad8-17f1-43c0-8faa-3f6e24d37ec0 admin admin] RBAC: Authorizing `identity:delete_user(user_id=6f1e736fab4f47ac868d1bed7400c571)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:03:59.735787 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c087fad8-17f1-43c0-8faa-3f6e24d37ec0 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:03:59.764210 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1105/2218] 10.222.0.27 () {58 vars in 958 bytes} [Tue Aug 30 02:03:59 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 30 02:03:59.774578 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-100967ec-331e-441b-8261-8cad53b12f6a None None] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:03:59.775050 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-100967ec-331e-441b-8261-8cad53b12f6a None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:03:59.775330 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-100967ec-331e-441b-8261-8cad53b12f6a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:03:59.796122 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.auth.core [None req-100967ec-331e-441b-8261-8cad53b12f6a None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81578) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:03:59.797310 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-100967ec-331e-441b-8261-8cad53b12f6a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:03:59.805205 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1106/2219] 10.222.0.27 () {58 vars in 958 bytes} [Tue Aug 30 02:03:59 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 02:03:59.809271 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-100967ec-331e-441b-8261-8cad53b12f6a None None] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:03:59.817924 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-100967ec-331e-441b-8261-8cad53b12f6a None None] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:03:59.823301 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7392d91c-1087-4cc8-a23f-23bf021e6667 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:03:59.823877 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7392d91c-1087-4cc8-a23f-23bf021e6667 admin admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:03:59.824152 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7392d91c-1087-4cc8-a23f-23bf021e6667 admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:03:59.824417 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7392d91c-1087-4cc8-a23f-23bf021e6667 admin admin] PATH_INFO: `/v3/domains` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:03:59.825047 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7392d91c-1087-4cc8-a23f-23bf021e6667 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81578) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 02:03:59.825345 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7392d91c-1087-4cc8-a23f-23bf021e6667 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:03:59.827010 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7392d91c-1087-4cc8-a23f-23bf021e6667 admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:03:59.839665 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1107/2220] 10.222.0.27 () {58 vars in 1153 bytes} [Tue Aug 30 02:03:59 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 30 02:03:59.846396 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7392d91c-1087-4cc8-a23f-23bf021e6667 admin admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:03:59.856352 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7392d91c-1087-4cc8-a23f-23bf021e6667 admin admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:03:59.861071 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0590b2f-13ad-47ff-89a5-4294fd6f1fd8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:03:59.861698 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0590b2f-13ad-47ff-89a5-4294fd6f1fd8 admin admin] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:03:59.861980 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0590b2f-13ad-47ff-89a5-4294fd6f1fd8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:03:59.862291 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0590b2f-13ad-47ff-89a5-4294fd6f1fd8 admin admin] PATH_INFO: `/v3/projects` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:03:59.866884 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0590b2f-13ad-47ff-89a5-4294fd6f1fd8 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:03:59.866884 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0590b2f-13ad-47ff-89a5-4294fd6f1fd8 admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:03:59.960223 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-c087fad8-17f1-43c0-8faa-3f6e24d37ec0 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '6f1e736fab4f47ac868d1bed7400c571'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:03:59.960927 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-c087fad8-17f1-43c0-8faa-3f6e24d37ec0 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '6f1e736fab4f47ac868d1bed7400c571'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:03:59.961570 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-c087fad8-17f1-43c0-8faa-3f6e24d37ec0 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '6f1e736fab4f47ac868d1bed7400c571'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:03:59.973916 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-c087fad8-17f1-43c0-8faa-3f6e24d37ec0 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '6f1e736fab4f47ac868d1bed7400c571'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:03:59.980661 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-c087fad8-17f1-43c0-8faa-3f6e24d37ec0 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '6f1e736fab4f47ac868d1bed7400c571'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:04:00.002796 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1108/2221] 10.222.0.27 () {60 vars in 1153 bytes} [Tue Aug 30 02:03:59 2022] POST /identity/v3/projects => generated 373 bytes in 154 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 02:04:00.004276 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0590b2f-13ad-47ff-89a5-4294fd6f1fd8 admin admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:04:00.006458 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-d0590b2f-13ad-47ff-89a5-4294fd6f1fd8 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:04:00.083687 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0590b2f-13ad-47ff-89a5-4294fd6f1fd8 admin admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:04:00.088696 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50c73562-68a7-45b2-9c98-140ecdbaf25b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:04:00.089611 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50c73562-68a7-45b2-9c98-140ecdbaf25b admin admin] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:04:00.089868 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50c73562-68a7-45b2-9c98-140ecdbaf25b admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:04:00.090605 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50c73562-68a7-45b2-9c98-140ecdbaf25b admin admin] PATH_INFO: `/v3/users` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:04:00.091657 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50c73562-68a7-45b2-9c98-140ecdbaf25b admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:04:00.093568 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50c73562-68a7-45b2-9c98-140ecdbaf25b admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:04:00.095123 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1114/2222] 10.222.0.27 () {58 vars in 1224 bytes} [Tue Aug 30 02:03:59 2022] DELETE /identity/v3/users/6f1e736fab4f47ac868d1bed7400c571 => generated 0 bytes in 455 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 02:04:00.099638 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e06a4037-c059-4ecc-a42a-dcb17ec005b1 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:04:00.099990 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e06a4037-c059-4ecc-a42a-dcb17ec005b1 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:04:00.100265 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e06a4037-c059-4ecc-a42a-dcb17ec005b1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:04:00.120322 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-e06a4037-c059-4ecc-a42a-dcb17ec005b1 None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:04:00.121558 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-e06a4037-c059-4ecc-a42a-dcb17ec005b1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:04:00.180308 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1115/2223] 10.222.0.27 () {58 vars in 958 bytes} [Tue Aug 30 02:04:00 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 81 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 02:04:00.190438 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1109/2224] 10.222.0.27 () {60 vars in 1144 bytes} [Tue Aug 30 02:04:00 2022] POST /identity/v3/users => generated 287 bytes in 184 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 02:04:00.191980 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e06a4037-c059-4ecc-a42a-dcb17ec005b1 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:04:00.193339 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50c73562-68a7-45b2-9c98-140ecdbaf25b admin admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:04:00.195791 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-50c73562-68a7-45b2-9c98-140ecdbaf25b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:04:00.204502 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e06a4037-c059-4ecc-a42a-dcb17ec005b1 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:04:00.209248 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16bb7e19-517f-49b4-b824-172f65470170 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:04:00.209941 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16bb7e19-517f-49b4-b824-172f65470170 admin admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:04:00.210275 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16bb7e19-517f-49b4-b824-172f65470170 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:04:00.210625 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16bb7e19-517f-49b4-b824-172f65470170 admin admin] PATH_INFO: `/v3/roles` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:04:00.211388 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16bb7e19-517f-49b4-b824-172f65470170 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:04:00.213068 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16bb7e19-517f-49b4-b824-172f65470170 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:04:00.223397 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1116/2225] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Aug 30 02:04:00 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 02:04:00.233359 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16bb7e19-517f-49b4-b824-172f65470170 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:04:00.245629 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16bb7e19-517f-49b4-b824-172f65470170 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:04:00.252193 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c450535-d765-4c9a-a532-64eb945e93fe admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:04:00.253032 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c450535-d765-4c9a-a532-64eb945e93fe admin admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:04:00.253377 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c450535-d765-4c9a-a532-64eb945e93fe admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:04:00.253741 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c450535-d765-4c9a-a532-64eb945e93fe admin admin] PATH_INFO: `/v3/roles` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:04:00.254590 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c450535-d765-4c9a-a532-64eb945e93fe admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:04:00.257128 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c450535-d765-4c9a-a532-64eb945e93fe admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:04:00.257420 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50c73562-68a7-45b2-9c98-140ecdbaf25b admin admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:04:00.269843 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1117/2226] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Aug 30 02:04:00 2022] GET /identity/v3/roles => generated 1420 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 30 02:04:00.274077 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c450535-d765-4c9a-a532-64eb945e93fe admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:04:00.289668 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c450535-d765-4c9a-a532-64eb945e93fe admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:04:00.295494 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49f6f46a-9a0f-402b-94ba-c2dc23d0b7d0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:04:00.296225 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49f6f46a-9a0f-402b-94ba-c2dc23d0b7d0 admin admin] REQUEST_METHOD: `PUT` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:04:00.296726 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49f6f46a-9a0f-402b-94ba-c2dc23d0b7d0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:04:00.297306 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49f6f46a-9a0f-402b-94ba-c2dc23d0b7d0 admin admin] PATH_INFO: `/v3/projects/463ec659a3f44218bd37b336608955e9/users/d5279d5735a04a7e8a9bda528fe57c56/roles/1fb81afcb9f245f3b989add94598a063` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:04:00.314223 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-250461d7-ef0b-4da8-aba8-ef382bc22b4a service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f3ea1580c56f4814a92bbb2acf542905', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4562f3951fbe465280073e60a11e44e2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:04:00.314845 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-250461d7-ef0b-4da8-aba8-ef382bc22b4a service neutron] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:04:00.315119 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-250461d7-ef0b-4da8-aba8-ef382bc22b4a service neutron] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:04:00.315563 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-250461d7-ef0b-4da8-aba8-ef382bc22b4a service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:04:00.319878 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49f6f46a-9a0f-402b-94ba-c2dc23d0b7d0 admin admin] RBAC: Authorizing `identity:create_grant(project_id=463ec659a3f44218bd37b336608955e9, user_id=d5279d5735a04a7e8a9bda528fe57c56, role_id=1fb81afcb9f245f3b989add94598a063)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:04:00.322815 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49f6f46a-9a0f-402b-94ba-c2dc23d0b7d0 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:04:00.326572 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-250461d7-ef0b-4da8-aba8-ef382bc22b4a service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:04:00.328343 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-250461d7-ef0b-4da8-aba8-ef382bc22b4a service neutron] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:04:00.342549 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1110/2227] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Aug 30 02:04:00 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 150 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 30 02:04:00.432081 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=87,keystone:DELETE=11,keystone:INSERT=2 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:04:00.439850 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1118/2228] 10.222.0.27 () {60 vars in 1483 bytes} [Tue Aug 30 02:04:00 2022] PUT /identity/v3/projects/463ec659a3f44218bd37b336608955e9/users/d5279d5735a04a7e8a9bda528fe57c56/roles/1fb81afcb9f245f3b989add94598a063 => generated 0 bytes in 166 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 02:04:00.450809 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d926a56-5941-4523-bcfd-f3bdc08f8ff3 None None] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:04:00.451404 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d926a56-5941-4523-bcfd-f3bdc08f8ff3 None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:04:00.451888 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d926a56-5941-4523-bcfd-f3bdc08f8ff3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:04:00.483514 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.auth.core [None req-7d926a56-5941-4523-bcfd-f3bdc08f8ff3 None None] MFA Rules not processed for user `d5279d5735a04a7e8a9bda528fe57c56`. Rule list: `[]` (Enabled: `True`). {{(pid=81578) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:04:00.484795 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-7d926a56-5941-4523-bcfd-f3bdc08f8ff3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:04:00.555751 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1111/2229] 10.222.0.27 () {58 vars in 958 bytes} [Tue Aug 30 02:04:00 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 02:04:00.592123 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:04:00.845273 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49f6f46a-9a0f-402b-94ba-c2dc23d0b7d0 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:04:00.848501 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-49f6f46a-9a0f-402b-94ba-c2dc23d0b7d0 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:04:00.909590 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49f6f46a-9a0f-402b-94ba-c2dc23d0b7d0 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:04:00.913416 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b0f829a-fdbb-4345-bb53-47a0b1539696 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ab936b3767a8417aa80c8c8f1b84b667', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4562f3951fbe465280073e60a11e44e2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:04:00.914109 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b0f829a-fdbb-4345-bb53-47a0b1539696 service cinder] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:04:00.914393 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b0f829a-fdbb-4345-bb53-47a0b1539696 service cinder] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:04:00.914665 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b0f829a-fdbb-4345-bb53-47a0b1539696 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:04:00.923511 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b0f829a-fdbb-4345-bb53-47a0b1539696 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:04:00.924920 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b0f829a-fdbb-4345-bb53-47a0b1539696 service cinder] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:04:00.935131 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1119/2230] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Aug 30 02:04:00 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 91 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 02:04:00.962625 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d926a56-5941-4523-bcfd-f3bdc08f8ff3 None None] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:04:01.027610 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b0f829a-fdbb-4345-bb53-47a0b1539696 service cinder] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:04:01.029261 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d926a56-5941-4523-bcfd-f3bdc08f8ff3 None None] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:04:01.029419 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-8b0f829a-fdbb-4345-bb53-47a0b1539696 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:04:01.034680 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96b29900-ebde-4af5-b50c-201675aba4ac admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:04:01.035736 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96b29900-ebde-4af5-b50c-201675aba4ac admin admin] REQUEST_METHOD: `DELETE` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:04:01.036013 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96b29900-ebde-4af5-b50c-201675aba4ac admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:04:01.036434 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96b29900-ebde-4af5-b50c-201675aba4ac admin admin] PATH_INFO: `/v3/users/d5279d5735a04a7e8a9bda528fe57c56` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:04:01.038825 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96b29900-ebde-4af5-b50c-201675aba4ac admin admin] RBAC: Authorizing `identity:delete_user(user_id=d5279d5735a04a7e8a9bda528fe57c56)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:04:01.041902 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96b29900-ebde-4af5-b50c-201675aba4ac admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:04:01.077627 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b0f829a-fdbb-4345-bb53-47a0b1539696 service cinder] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:04:01.088177 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d26eaffb-b7e1-4195-9708-93fd2c27fe54 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:04:01.088834 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d26eaffb-b7e1-4195-9708-93fd2c27fe54 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:04:01.089177 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d26eaffb-b7e1-4195-9708-93fd2c27fe54 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:04:01.089483 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d26eaffb-b7e1-4195-9708-93fd2c27fe54 admin admin] PATH_INFO: `/v3/projects/8527acd3c1964f7f9d67b413bf9ef488` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:04:01.090633 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d26eaffb-b7e1-4195-9708-93fd2c27fe54 admin admin] RBAC: Authorizing `identity:delete_project(project_id=8527acd3c1964f7f9d67b413bf9ef488)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:04:01.093026 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d26eaffb-b7e1-4195-9708-93fd2c27fe54 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:04:01.359455 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-96b29900-ebde-4af5-b50c-201675aba4ac admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'd5279d5735a04a7e8a9bda528fe57c56'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:04:01.359665 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-d26eaffb-b7e1-4195-9708-93fd2c27fe54 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '8527acd3c1964f7f9d67b413bf9ef488', 'request_id': 'req-d26eaffb-b7e1-4195-9708-93fd2c27fe54'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:04:01.360126 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-96b29900-ebde-4af5-b50c-201675aba4ac admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'd5279d5735a04a7e8a9bda528fe57c56'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:04:01.365030 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-d26eaffb-b7e1-4195-9708-93fd2c27fe54 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '8527acd3c1964f7f9d67b413bf9ef488', 'request_id': 'req-d26eaffb-b7e1-4195-9708-93fd2c27fe54'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:04:01.371898 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-d26eaffb-b7e1-4195-9708-93fd2c27fe54 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '8527acd3c1964f7f9d67b413bf9ef488', 'request_id': 'req-d26eaffb-b7e1-4195-9708-93fd2c27fe54'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:04:01.477201 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-96b29900-ebde-4af5-b50c-201675aba4ac admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'd5279d5735a04a7e8a9bda528fe57c56'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:04:01.477843 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-96b29900-ebde-4af5-b50c-201675aba4ac admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'd5279d5735a04a7e8a9bda528fe57c56'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:04:01.485290 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-96b29900-ebde-4af5-b50c-201675aba4ac admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'd5279d5735a04a7e8a9bda528fe57c56'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:04:01.502907 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1112/2231] 10.222.0.27 () {58 vars in 1224 bytes} [Tue Aug 30 02:04:00 2022] DELETE /identity/v3/users/d5279d5735a04a7e8a9bda528fe57c56 => generated 0 bytes in 541 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 02:04:01.506301 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:04:01.508173 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-281bfcdc-72a5-4ef2-8cb7-1ef4e5ca1650 None None] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:04:01.508467 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-281bfcdc-72a5-4ef2-8cb7-1ef4e5ca1650 None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:04:01.508840 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-281bfcdc-72a5-4ef2-8cb7-1ef4e5ca1650 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:04:01.528795 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.auth.core [None req-281bfcdc-72a5-4ef2-8cb7-1ef4e5ca1650 None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81578) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:04:01.530111 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-281bfcdc-72a5-4ef2-8cb7-1ef4e5ca1650 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:04:01.592900 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1113/2232] 10.222.0.27 () {58 vars in 958 bytes} [Tue Aug 30 02:04:01 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 30 02:04:01.604011 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-281bfcdc-72a5-4ef2-8cb7-1ef4e5ca1650 None None] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:04:01.605412 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-281bfcdc-72a5-4ef2-8cb7-1ef4e5ca1650 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:04:01.614825 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-d26eaffb-b7e1-4195-9708-93fd2c27fe54 admin admin] The token cache is being invalidate because project 8527acd3c1964f7f9d67b413bf9ef488 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81577) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 30 02:04:01.615209 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-d26eaffb-b7e1-4195-9708-93fd2c27fe54 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:04:01.615915 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-d26eaffb-b7e1-4195-9708-93fd2c27fe54 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:04:01.623086 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1120/2233] 10.222.0.27 () {58 vars in 1233 bytes} [Tue Aug 30 02:04:01 2022] DELETE /identity/v3/projects/8527acd3c1964f7f9d67b413bf9ef488 => generated 0 bytes in 592 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 30 02:04:01.625678 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d965b0dc-de0d-4b81-9df5-e1b19e217a9a None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:04:01.625967 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d965b0dc-de0d-4b81-9df5-e1b19e217a9a None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:04:01.626305 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d965b0dc-de0d-4b81-9df5-e1b19e217a9a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:04:01.648003 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-d965b0dc-de0d-4b81-9df5-e1b19e217a9a None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:04:01.649242 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-d965b0dc-de0d-4b81-9df5-e1b19e217a9a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:04:01.658848 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1121/2234] 10.222.0.27 () {58 vars in 958 bytes} [Tue Aug 30 02:04:01 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 02:04:01.666306 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09ce1ce7-12f1-42da-85e9-3f1cdc6c9470 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:04:01.666639 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09ce1ce7-12f1-42da-85e9-3f1cdc6c9470 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:04:01.666968 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09ce1ce7-12f1-42da-85e9-3f1cdc6c9470 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:04:01.675922 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-281bfcdc-72a5-4ef2-8cb7-1ef4e5ca1650 None None] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:04:01.678297 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7c2e916-995c-416c-a8b3-c0d0712be71e service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f3ea1580c56f4814a92bbb2acf542905', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4562f3951fbe465280073e60a11e44e2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:04:01.678952 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7c2e916-995c-416c-a8b3-c0d0712be71e service neutron] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:04:01.679935 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7c2e916-995c-416c-a8b3-c0d0712be71e service neutron] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:04:01.679935 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7c2e916-995c-416c-a8b3-c0d0712be71e service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:04:01.681605 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-d7c2e916-995c-416c-a8b3-c0d0712be71e service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:04:01.686803 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-09ce1ce7-12f1-42da-85e9-3f1cdc6c9470 None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:04:01.687981 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-09ce1ce7-12f1-42da-85e9-3f1cdc6c9470 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:04:01.700608 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1122/2235] 10.222.0.27 () {58 vars in 958 bytes} [Tue Aug 30 02:04:01 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 35 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 30 02:04:01.700922 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7c2e916-995c-416c-a8b3-c0d0712be71e service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:04:01.705310 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7c2e916-995c-416c-a8b3-c0d0712be71e service neutron] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:04:01.708763 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09ce1ce7-12f1-42da-85e9-3f1cdc6c9470 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:04:01.714080 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1114/2236] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Aug 30 02:04:01 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 111 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 30 02:04:01.718567 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09ce1ce7-12f1-42da-85e9-3f1cdc6c9470 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:04:01.723529 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6564cca-1726-4bb1-9913-7ec94f5452b7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:04:01.727845 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6564cca-1726-4bb1-9913-7ec94f5452b7 admin admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:04:01.727845 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6564cca-1726-4bb1-9913-7ec94f5452b7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:04:01.727845 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6564cca-1726-4bb1-9913-7ec94f5452b7 admin admin] PATH_INFO: `/v3/domains` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:04:01.727845 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6564cca-1726-4bb1-9913-7ec94f5452b7 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 02:04:01.727845 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6564cca-1726-4bb1-9913-7ec94f5452b7 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:04:01.729137 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6564cca-1726-4bb1-9913-7ec94f5452b7 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:04:01.740651 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1123/2237] 10.222.0.27 () {58 vars in 1153 bytes} [Tue Aug 30 02:04:01 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 30 02:04:01.745946 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7c2e916-995c-416c-a8b3-c0d0712be71e service neutron] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:04:01.756042 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7c2e916-995c-416c-a8b3-c0d0712be71e service neutron] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:04:01.761829 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc180701-894c-45e9-b7eb-b152aa710a8b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:04:01.762449 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc180701-894c-45e9-b7eb-b152aa710a8b admin admin] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:04:01.762728 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc180701-894c-45e9-b7eb-b152aa710a8b admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:04:01.762997 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc180701-894c-45e9-b7eb-b152aa710a8b admin admin] PATH_INFO: `/v3/projects` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:04:01.763633 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc180701-894c-45e9-b7eb-b152aa710a8b admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:04:01.765573 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc180701-894c-45e9-b7eb-b152aa710a8b admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:04:01.849490 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:04:01.857789 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1115/2238] 10.222.0.27 () {60 vars in 1153 bytes} [Tue Aug 30 02:04:01 2022] POST /identity/v3/projects => generated 373 bytes in 112 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 02:04:01.861024 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6564cca-1726-4bb1-9913-7ec94f5452b7 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:04:01.871170 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6564cca-1726-4bb1-9913-7ec94f5452b7 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:04:01.930396 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69023901-bfe0-4d69-ac0f-c9114ab77626 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:04:01.930999 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69023901-bfe0-4d69-ac0f-c9114ab77626 admin admin] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:04:01.931282 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69023901-bfe0-4d69-ac0f-c9114ab77626 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:04:01.931558 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69023901-bfe0-4d69-ac0f-c9114ab77626 admin admin] PATH_INFO: `/v3/users` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:04:01.932194 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69023901-bfe0-4d69-ac0f-c9114ab77626 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:04:01.937937 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69023901-bfe0-4d69-ac0f-c9114ab77626 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:04:02.096365 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1124/2239] 10.222.0.27 () {60 vars in 1144 bytes} [Tue Aug 30 02:04:01 2022] POST /identity/v3/users => generated 287 bytes in 233 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 02:04:02.097905 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc180701-894c-45e9-b7eb-b152aa710a8b admin admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:04:02.113192 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc180701-894c-45e9-b7eb-b152aa710a8b admin admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:04:02.120582 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95e7e472-0cfe-4cab-98e2-b3efea2bda11 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:04:02.121503 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95e7e472-0cfe-4cab-98e2-b3efea2bda11 admin admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:04:02.122758 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95e7e472-0cfe-4cab-98e2-b3efea2bda11 admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:04:02.122758 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95e7e472-0cfe-4cab-98e2-b3efea2bda11 admin admin] PATH_INFO: `/v3/roles` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:04:02.123583 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95e7e472-0cfe-4cab-98e2-b3efea2bda11 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:04:02.125382 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95e7e472-0cfe-4cab-98e2-b3efea2bda11 admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:04:02.141420 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1116/2240] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Aug 30 02:04:02 2022] GET /identity/v3/roles => generated 1420 bytes in 45 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 02:04:02.146720 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69023901-bfe0-4d69-ac0f-c9114ab77626 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:04:02.157694 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69023901-bfe0-4d69-ac0f-c9114ab77626 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:04:02.162869 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-371d60a2-11b5-47b6-b0cf-b167f0cc6e07 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:04:02.163608 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-371d60a2-11b5-47b6-b0cf-b167f0cc6e07 admin admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:04:02.163893 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-371d60a2-11b5-47b6-b0cf-b167f0cc6e07 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:04:02.164142 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-371d60a2-11b5-47b6-b0cf-b167f0cc6e07 admin admin] PATH_INFO: `/v3/roles` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:04:02.164786 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-371d60a2-11b5-47b6-b0cf-b167f0cc6e07 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:04:02.166408 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-371d60a2-11b5-47b6-b0cf-b167f0cc6e07 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:04:02.179672 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1125/2241] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Aug 30 02:04:02 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 02:04:02.183884 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95e7e472-0cfe-4cab-98e2-b3efea2bda11 admin admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:04:02.203165 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95e7e472-0cfe-4cab-98e2-b3efea2bda11 admin admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:04:02.205602 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab74bd98-81f1-48e0-9cbf-d84ae1c2d18e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:04:02.206426 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab74bd98-81f1-48e0-9cbf-d84ae1c2d18e admin admin] REQUEST_METHOD: `PUT` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:04:02.206585 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab74bd98-81f1-48e0-9cbf-d84ae1c2d18e admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:04:02.207077 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab74bd98-81f1-48e0-9cbf-d84ae1c2d18e admin admin] PATH_INFO: `/v3/projects/d779875224e742f9a3f3bf4bfcf12fbc/users/dbce595e3bec45deb120a3d6fe6d6091/roles/1fb81afcb9f245f3b989add94598a063` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:04:02.225043 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab74bd98-81f1-48e0-9cbf-d84ae1c2d18e admin admin] RBAC: Authorizing `identity:create_grant(project_id=d779875224e742f9a3f3bf4bfcf12fbc, user_id=dbce595e3bec45deb120a3d6fe6d6091, role_id=1fb81afcb9f245f3b989add94598a063)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:04:02.228804 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab74bd98-81f1-48e0-9cbf-d84ae1c2d18e admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:04:02.276599 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-371d60a2-11b5-47b6-b0cf-b167f0cc6e07 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:04:02.278472 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-371d60a2-11b5-47b6-b0cf-b167f0cc6e07 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:04:02.300196 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-371d60a2-11b5-47b6-b0cf-b167f0cc6e07 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:04:02.305314 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d7ebc3c-cf36-4160-aeed-4464af7b7f06 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:04:02.305962 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d7ebc3c-cf36-4160-aeed-4464af7b7f06 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:04:02.306230 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d7ebc3c-cf36-4160-aeed-4464af7b7f06 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:04:02.306551 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d7ebc3c-cf36-4160-aeed-4464af7b7f06 admin admin] PATH_INFO: `/v3/projects/463ec659a3f44218bd37b336608955e9` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:04:02.310519 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d7ebc3c-cf36-4160-aeed-4464af7b7f06 admin admin] RBAC: Authorizing `identity:delete_project(project_id=463ec659a3f44218bd37b336608955e9)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:04:02.312531 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d7ebc3c-cf36-4160-aeed-4464af7b7f06 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:04:02.359840 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1117/2242] 10.222.0.27 () {60 vars in 1483 bytes} [Tue Aug 30 02:04:02 2022] PUT /identity/v3/projects/d779875224e742f9a3f3bf4bfcf12fbc/users/dbce595e3bec45deb120a3d6fe6d6091/roles/1fb81afcb9f245f3b989add94598a063 => generated 0 bytes in 177 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 30 02:04:02.371360 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63831c6c-3b84-438c-b0f4-9537fb758ca9 None None] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:04:02.371576 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63831c6c-3b84-438c-b0f4-9537fb758ca9 None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:04:02.371853 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63831c6c-3b84-438c-b0f4-9537fb758ca9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:04:02.400450 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.auth.core [None req-63831c6c-3b84-438c-b0f4-9537fb758ca9 None None] MFA Rules not processed for user `dbce595e3bec45deb120a3d6fe6d6091`. Rule list: `[]` (Enabled: `True`). {{(pid=81578) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:04:02.401725 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-63831c6c-3b84-438c-b0f4-9537fb758ca9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:04:02.477570 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1118/2243] 10.222.0.27 () {58 vars in 958 bytes} [Tue Aug 30 02:04:02 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 30 02:04:02.488424 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-7d7ebc3c-cf36-4160-aeed-4464af7b7f06 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '463ec659a3f44218bd37b336608955e9', 'request_id': 'req-7d7ebc3c-cf36-4160-aeed-4464af7b7f06'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:04:02.495660 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-7d7ebc3c-cf36-4160-aeed-4464af7b7f06 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '463ec659a3f44218bd37b336608955e9', 'request_id': 'req-7d7ebc3c-cf36-4160-aeed-4464af7b7f06'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:04:02.502759 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-7d7ebc3c-cf36-4160-aeed-4464af7b7f06 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '463ec659a3f44218bd37b336608955e9', 'request_id': 'req-7d7ebc3c-cf36-4160-aeed-4464af7b7f06'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:04:02.505428 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63831c6c-3b84-438c-b0f4-9537fb758ca9 None None] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:04:02.508071 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-63831c6c-3b84-438c-b0f4-9537fb758ca9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:04:02.571107 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63831c6c-3b84-438c-b0f4-9537fb758ca9 None None] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:04:02.613520 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-7d7ebc3c-cf36-4160-aeed-4464af7b7f06 admin admin] The token cache is being invalidate because project 463ec659a3f44218bd37b336608955e9 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81577) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 30 02:04:02.613825 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-7d7ebc3c-cf36-4160-aeed-4464af7b7f06 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:04:02.614554 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-7d7ebc3c-cf36-4160-aeed-4464af7b7f06 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:04:02.616378 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1126/2244] 10.222.0.27 () {58 vars in 1233 bytes} [Tue Aug 30 02:04:02 2022] DELETE /identity/v3/projects/463ec659a3f44218bd37b336608955e9 => generated 0 bytes in 340 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 30 02:04:02.620848 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a25e3816-73ee-4e72-88c6-fe4242810650 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:04:02.621172 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a25e3816-73ee-4e72-88c6-fe4242810650 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:04:02.621548 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a25e3816-73ee-4e72-88c6-fe4242810650 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:04:02.621754 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b77c998d-2f27-4898-a302-422b45efd820 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ab936b3767a8417aa80c8c8f1b84b667', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4562f3951fbe465280073e60a11e44e2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:04:02.622366 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b77c998d-2f27-4898-a302-422b45efd820 service cinder] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:04:02.622647 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b77c998d-2f27-4898-a302-422b45efd820 service cinder] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:04:02.622921 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b77c998d-2f27-4898-a302-422b45efd820 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:04:02.624749 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-b77c998d-2f27-4898-a302-422b45efd820 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:04:02.642970 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-a25e3816-73ee-4e72-88c6-fe4242810650 None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:04:02.644184 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-a25e3816-73ee-4e72-88c6-fe4242810650 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:04:02.688623 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b77c998d-2f27-4898-a302-422b45efd820 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:04:02.690116 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b77c998d-2f27-4898-a302-422b45efd820 service cinder] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:04:02.700074 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1119/2245] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Aug 30 02:04:02 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 195 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 30 02:04:02.711412 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1127/2246] 10.222.0.27 () {58 vars in 958 bytes} [Tue Aug 30 02:04:02 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 30 02:04:02.712993 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc733b07-67b6-4f14-86e3-af7661c7875a None None] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:04:02.713287 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc733b07-67b6-4f14-86e3-af7661c7875a None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:04:02.713585 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc733b07-67b6-4f14-86e3-af7661c7875a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:04:02.738369 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.auth.core [None req-cc733b07-67b6-4f14-86e3-af7661c7875a None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81578) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:04:02.739666 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-cc733b07-67b6-4f14-86e3-af7661c7875a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:04:02.751382 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1120/2247] 10.222.0.27 () {58 vars in 958 bytes} [Tue Aug 30 02:04:02 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 38 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 02:04:02.757510 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a25e3816-73ee-4e72-88c6-fe4242810650 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:04:02.777369 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a25e3816-73ee-4e72-88c6-fe4242810650 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:04:02.796933 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e421233-59cf-498f-ae02-7566c7912957 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:04:02.798030 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e421233-59cf-498f-ae02-7566c7912957 admin admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:04:02.798746 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e421233-59cf-498f-ae02-7566c7912957 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:04:02.799088 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e421233-59cf-498f-ae02-7566c7912957 admin admin] PATH_INFO: `/v3/domains` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:04:02.800094 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e421233-59cf-498f-ae02-7566c7912957 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 02:04:02.800435 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e421233-59cf-498f-ae02-7566c7912957 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:04:02.802358 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e421233-59cf-498f-ae02-7566c7912957 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:04:02.818957 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1128/2248] 10.222.0.27 () {58 vars in 1153 bytes} [Tue Aug 30 02:04:02 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 63 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 02:04:02.844735 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc733b07-67b6-4f14-86e3-af7661c7875a None None] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:04:02.844735 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc733b07-67b6-4f14-86e3-af7661c7875a None None] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:04:02.854557 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ee6a039-b03f-4d0f-9a9e-9cf0f63f41c9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:04:02.855435 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ee6a039-b03f-4d0f-9a9e-9cf0f63f41c9 admin admin] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:04:02.855825 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ee6a039-b03f-4d0f-9a9e-9cf0f63f41c9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:04:02.856599 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ee6a039-b03f-4d0f-9a9e-9cf0f63f41c9 admin admin] PATH_INFO: `/v3/projects` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:04:02.856876 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ee6a039-b03f-4d0f-9a9e-9cf0f63f41c9 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:04:02.867452 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ee6a039-b03f-4d0f-9a9e-9cf0f63f41c9 admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:04:03.025332 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1121/2249] 10.222.0.27 () {60 vars in 1153 bytes} [Tue Aug 30 02:04:02 2022] POST /identity/v3/projects => generated 389 bytes in 198 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 02:04:03.026615 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e421233-59cf-498f-ae02-7566c7912957 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:04:03.036797 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e421233-59cf-498f-ae02-7566c7912957 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:04:03.094737 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb626235-2870-430c-9816-5bfc037d02d4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:04:03.095335 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb626235-2870-430c-9816-5bfc037d02d4 admin admin] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:04:03.095611 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb626235-2870-430c-9816-5bfc037d02d4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:04:03.095876 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb626235-2870-430c-9816-5bfc037d02d4 admin admin] PATH_INFO: `/v3/users` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:04:03.096495 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb626235-2870-430c-9816-5bfc037d02d4 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:04:03.098586 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb626235-2870-430c-9816-5bfc037d02d4 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:04:03.106029 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:04:03.180901 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:INSERT=5,keystone:SELECT=95 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:04:03.338692 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1129/2250] 10.222.0.27 () {60 vars in 1144 bytes} [Tue Aug 30 02:04:03 2022] POST /identity/v3/users => generated 295 bytes in 309 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 02:04:03.339361 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ee6a039-b03f-4d0f-9a9e-9cf0f63f41c9 admin admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:04:03.350246 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ee6a039-b03f-4d0f-9a9e-9cf0f63f41c9 admin admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:04:03.355885 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55b6b9e4-6a55-4171-be69-a3c6a7da679f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:04:03.356433 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55b6b9e4-6a55-4171-be69-a3c6a7da679f admin admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:04:03.357202 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb626235-2870-430c-9816-5bfc037d02d4 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:04:03.357354 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55b6b9e4-6a55-4171-be69-a3c6a7da679f admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:04:03.357649 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55b6b9e4-6a55-4171-be69-a3c6a7da679f admin admin] PATH_INFO: `/v3/roles` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:04:03.359274 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55b6b9e4-6a55-4171-be69-a3c6a7da679f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:04:03.360586 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-bb626235-2870-430c-9816-5bfc037d02d4 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:04:03.360893 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55b6b9e4-6a55-4171-be69-a3c6a7da679f admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:04:03.372912 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1122/2251] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Aug 30 02:04:03 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 02:04:03.377665 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55b6b9e4-6a55-4171-be69-a3c6a7da679f admin admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:04:03.381110 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb626235-2870-430c-9816-5bfc037d02d4 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:04:03.387470 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40f41723-62dd-48b6-906e-f4d8d247d876 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:04:03.388413 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55b6b9e4-6a55-4171-be69-a3c6a7da679f admin admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:04:03.400028 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40f41723-62dd-48b6-906e-f4d8d247d876 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:04:03.400028 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40f41723-62dd-48b6-906e-f4d8d247d876 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:04:03.400028 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40f41723-62dd-48b6-906e-f4d8d247d876 admin admin] PATH_INFO: `/v3/users/b3b5830806e643a1a7577a11a66c554d` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:04:03.400028 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40f41723-62dd-48b6-906e-f4d8d247d876 admin admin] RBAC: Authorizing `identity:delete_user(user_id=b3b5830806e643a1a7577a11a66c554d)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:04:03.400605 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ff5d9cf-3949-4cd7-9628-50c057a59c72 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:04:03.400710 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40f41723-62dd-48b6-906e-f4d8d247d876 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:04:03.400878 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ff5d9cf-3949-4cd7-9628-50c057a59c72 admin admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:04:03.400878 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ff5d9cf-3949-4cd7-9628-50c057a59c72 admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:04:03.400878 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ff5d9cf-3949-4cd7-9628-50c057a59c72 admin admin] PATH_INFO: `/v3/roles` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:04:03.400878 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ff5d9cf-3949-4cd7-9628-50c057a59c72 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:04:03.400878 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ff5d9cf-3949-4cd7-9628-50c057a59c72 admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:04:03.406953 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1123/2252] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Aug 30 02:04:03 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 02:04:03.413374 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ff5d9cf-3949-4cd7-9628-50c057a59c72 admin admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:04:03.423609 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ff5d9cf-3949-4cd7-9628-50c057a59c72 admin admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:04:03.428196 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6eaa36ba-804f-45f8-a57b-d86e2e57f719 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:04:03.429500 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6eaa36ba-804f-45f8-a57b-d86e2e57f719 admin admin] REQUEST_METHOD: `PUT` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:04:03.429770 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6eaa36ba-804f-45f8-a57b-d86e2e57f719 admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:04:03.430044 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6eaa36ba-804f-45f8-a57b-d86e2e57f719 admin admin] PATH_INFO: `/v3/projects/7ebe8393a20f421ba3a7e6f26d5d2696/users/cd95476d9b054391a02a08caa89238a4/roles/1fb81afcb9f245f3b989add94598a063` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:04:03.444400 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6eaa36ba-804f-45f8-a57b-d86e2e57f719 admin admin] RBAC: Authorizing `identity:create_grant(project_id=7ebe8393a20f421ba3a7e6f26d5d2696, user_id=cd95476d9b054391a02a08caa89238a4, role_id=1fb81afcb9f245f3b989add94598a063)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:04:03.447361 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6eaa36ba-804f-45f8-a57b-d86e2e57f719 admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:04:03.610571 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1124/2253] 10.222.0.27 () {60 vars in 1483 bytes} [Tue Aug 30 02:04:03 2022] PUT /identity/v3/projects/7ebe8393a20f421ba3a7e6f26d5d2696/users/cd95476d9b054391a02a08caa89238a4/roles/1fb81afcb9f245f3b989add94598a063 => generated 0 bytes in 200 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 30 02:04:03.622204 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a23cbff-9fdb-4e75-a816-888611a3aa54 None None] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:04:03.623156 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a23cbff-9fdb-4e75-a816-888611a3aa54 None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:04:03.623417 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a23cbff-9fdb-4e75-a816-888611a3aa54 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:04:03.651247 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.auth.core [None req-3a23cbff-9fdb-4e75-a816-888611a3aa54 None None] MFA Rules not processed for user `cd95476d9b054391a02a08caa89238a4`. Rule list: `[]` (Enabled: `True`). {{(pid=81578) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:04:03.652348 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-3a23cbff-9fdb-4e75-a816-888611a3aa54 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:04:03.725063 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1125/2254] 10.222.0.27 () {58 vars in 958 bytes} [Tue Aug 30 02:04:03 2022] POST /identity/v3/auth/tokens => generated 3266 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 30 02:04:03.744125 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-40f41723-62dd-48b6-906e-f4d8d247d876 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'b3b5830806e643a1a7577a11a66c554d'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:04:03.744622 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-40f41723-62dd-48b6-906e-f4d8d247d876 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'b3b5830806e643a1a7577a11a66c554d'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:04:03.745246 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-40f41723-62dd-48b6-906e-f4d8d247d876 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'b3b5830806e643a1a7577a11a66c554d'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:04:03.760476 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-40f41723-62dd-48b6-906e-f4d8d247d876 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'b3b5830806e643a1a7577a11a66c554d'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:04:03.769107 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-40f41723-62dd-48b6-906e-f4d8d247d876 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'b3b5830806e643a1a7577a11a66c554d'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:04:03.840107 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a23cbff-9fdb-4e75-a816-888611a3aa54 None None] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:04:03.841797 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-3a23cbff-9fdb-4e75-a816-888611a3aa54 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:04:03.856203 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1130/2255] 10.222.0.27 () {58 vars in 1224 bytes} [Tue Aug 30 02:04:03 2022] DELETE /identity/v3/users/b3b5830806e643a1a7577a11a66c554d => generated 0 bytes in 501 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 30 02:04:03.860614 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfa7d420-f651-4c3c-8996-30b03efbf977 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:04:03.860908 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfa7d420-f651-4c3c-8996-30b03efbf977 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:04:03.861177 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfa7d420-f651-4c3c-8996-30b03efbf977 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:04:03.880815 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-dfa7d420-f651-4c3c-8996-30b03efbf977 None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:04:03.882656 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-dfa7d420-f651-4c3c-8996-30b03efbf977 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:04:03.900596 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:04:03.922099 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a23cbff-9fdb-4e75-a816-888611a3aa54 None None] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:04:03.949088 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1131/2256] 10.222.0.27 () {58 vars in 958 bytes} [Tue Aug 30 02:04:03 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 02:04:03.960356 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfa7d420-f651-4c3c-8996-30b03efbf977 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:04:03.969373 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-dfa7d420-f651-4c3c-8996-30b03efbf977 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:04:03.974767 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d9e5362-de62-4029-82b8-06e76669b8f3 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ab936b3767a8417aa80c8c8f1b84b667', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4562f3951fbe465280073e60a11e44e2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:04:03.975865 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d9e5362-de62-4029-82b8-06e76669b8f3 service cinder] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:04:03.976362 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d9e5362-de62-4029-82b8-06e76669b8f3 service cinder] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:04:03.976722 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d9e5362-de62-4029-82b8-06e76669b8f3 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:04:03.978820 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-0d9e5362-de62-4029-82b8-06e76669b8f3 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:04:04.059265 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfa7d420-f651-4c3c-8996-30b03efbf977 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:04:04.059265 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbe0969c-bf31-4918-ba20-22b71e73fcf8 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f3ea1580c56f4814a92bbb2acf542905', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4562f3951fbe465280073e60a11e44e2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:04:04.059265 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbe0969c-bf31-4918-ba20-22b71e73fcf8 service neutron] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:04:04.059265 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbe0969c-bf31-4918-ba20-22b71e73fcf8 service neutron] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:04:04.060426 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbe0969c-bf31-4918-ba20-22b71e73fcf8 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:04:04.067052 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbe0969c-bf31-4918-ba20-22b71e73fcf8 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:04:04.068612 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbe0969c-bf31-4918-ba20-22b71e73fcf8 service neutron] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:04:04.073437 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d9e5362-de62-4029-82b8-06e76669b8f3 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:04:04.075137 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d9e5362-de62-4029-82b8-06e76669b8f3 service cinder] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:04:04.080028 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1132/2257] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Aug 30 02:04:03 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 121 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 02:04:04.084730 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1126/2258] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Aug 30 02:04:03 2022] GET /identity/v3/auth/tokens => generated 3266 bytes in 245 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 30 02:04:04.532253 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbe0969c-bf31-4918-ba20-22b71e73fcf8 service neutron] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:04:04.541006 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-bbe0969c-bf31-4918-ba20-22b71e73fcf8 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:04:04.550337 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbe0969c-bf31-4918-ba20-22b71e73fcf8 service neutron] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:04:04.555560 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bce9472-445b-4964-b7ec-54a0a6093d99 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:04:04.556192 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bce9472-445b-4964-b7ec-54a0a6093d99 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:04:04.556471 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bce9472-445b-4964-b7ec-54a0a6093d99 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:04:04.556741 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bce9472-445b-4964-b7ec-54a0a6093d99 admin admin] PATH_INFO: `/v3/users/a579fb8e4edd4faea19b9675c22ac4ab` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:04:04.557893 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bce9472-445b-4964-b7ec-54a0a6093d99 admin admin] RBAC: Authorizing `identity:delete_user(user_id=a579fb8e4edd4faea19b9675c22ac4ab)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:04:04.559782 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bce9472-445b-4964-b7ec-54a0a6093d99 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:04:04.564002 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=94,keystone:INSERT=6 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:04:04.654888 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d9e5362-de62-4029-82b8-06e76669b8f3 service cinder] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:04:04.661885 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-0d9e5362-de62-4029-82b8-06e76669b8f3 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:04:04.691894 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d9e5362-de62-4029-82b8-06e76669b8f3 service cinder] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:04:04.698221 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58f4bb8b-bb4c-48d9-a402-c731a32c9b0c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:04:04.699104 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58f4bb8b-bb4c-48d9-a402-c731a32c9b0c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:04:04.699548 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58f4bb8b-bb4c-48d9-a402-c731a32c9b0c admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:04:04.700014 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58f4bb8b-bb4c-48d9-a402-c731a32c9b0c admin admin] PATH_INFO: `/v3/projects/14567ca5cc3249f4aef186f05fad1cda` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:04:04.701412 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58f4bb8b-bb4c-48d9-a402-c731a32c9b0c admin admin] RBAC: Authorizing `identity:delete_project(project_id=14567ca5cc3249f4aef186f05fad1cda)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:04:04.709829 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58f4bb8b-bb4c-48d9-a402-c731a32c9b0c admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:04:04.821964 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-6bce9472-445b-4964-b7ec-54a0a6093d99 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'a579fb8e4edd4faea19b9675c22ac4ab'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:04:04.823521 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-6bce9472-445b-4964-b7ec-54a0a6093d99 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'a579fb8e4edd4faea19b9675c22ac4ab'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:04:04.824732 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-6bce9472-445b-4964-b7ec-54a0a6093d99 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'a579fb8e4edd4faea19b9675c22ac4ab'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:04:04.837264 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-6bce9472-445b-4964-b7ec-54a0a6093d99 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'a579fb8e4edd4faea19b9675c22ac4ab'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:04:04.844381 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-6bce9472-445b-4964-b7ec-54a0a6093d99 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'a579fb8e4edd4faea19b9675c22ac4ab'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:04:04.883202 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-58f4bb8b-bb4c-48d9-a402-c731a32c9b0c admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '14567ca5cc3249f4aef186f05fad1cda', 'request_id': 'req-58f4bb8b-bb4c-48d9-a402-c731a32c9b0c'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:04:04.888589 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-58f4bb8b-bb4c-48d9-a402-c731a32c9b0c admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '14567ca5cc3249f4aef186f05fad1cda', 'request_id': 'req-58f4bb8b-bb4c-48d9-a402-c731a32c9b0c'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:04:04.894855 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-58f4bb8b-bb4c-48d9-a402-c731a32c9b0c admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '14567ca5cc3249f4aef186f05fad1cda', 'request_id': 'req-58f4bb8b-bb4c-48d9-a402-c731a32c9b0c'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:04:04.964388 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1133/2259] 10.222.0.27 () {58 vars in 1224 bytes} [Tue Aug 30 02:04:04 2022] DELETE /identity/v3/users/a579fb8e4edd4faea19b9675c22ac4ab => generated 0 bytes in 432 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 30 02:04:04.969078 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db3bf5ef-9365-4dc6-b4c6-c09fc1da2746 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:04:04.969371 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db3bf5ef-9365-4dc6-b4c6-c09fc1da2746 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:04:04.969679 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db3bf5ef-9365-4dc6-b4c6-c09fc1da2746 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:04:04.988843 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-db3bf5ef-9365-4dc6-b4c6-c09fc1da2746 None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:04:04.990074 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-db3bf5ef-9365-4dc6-b4c6-c09fc1da2746 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:04:04.992896 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:04:05.025405 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-58f4bb8b-bb4c-48d9-a402-c731a32c9b0c admin admin] The token cache is being invalidate because project 14567ca5cc3249f4aef186f05fad1cda was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81578) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 30 02:04:05.025405 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-58f4bb8b-bb4c-48d9-a402-c731a32c9b0c admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:04:05.026946 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-58f4bb8b-bb4c-48d9-a402-c731a32c9b0c admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:04:05.028672 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1127/2260] 10.222.0.27 () {58 vars in 1233 bytes} [Tue Aug 30 02:04:04 2022] DELETE /identity/v3/projects/14567ca5cc3249f4aef186f05fad1cda => generated 0 bytes in 374 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 30 02:04:05.035031 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae1d133a-1407-479b-8608-b24b2ab5fe30 None None] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:04:05.035457 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae1d133a-1407-479b-8608-b24b2ab5fe30 None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:04:05.035789 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae1d133a-1407-479b-8608-b24b2ab5fe30 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:04:05.059068 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1134/2261] 10.222.0.27 () {58 vars in 958 bytes} [Tue Aug 30 02:04:04 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 30 02:04:05.059275 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.auth.core [None req-ae1d133a-1407-479b-8608-b24b2ab5fe30 None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81578) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:04:05.060934 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-ae1d133a-1407-479b-8608-b24b2ab5fe30 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:04:05.069063 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db3bf5ef-9365-4dc6-b4c6-c09fc1da2746 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:04:05.070536 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1128/2262] 10.222.0.27 () {58 vars in 958 bytes} [Tue Aug 30 02:04:05 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 37 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 30 02:04:05.070661 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-db3bf5ef-9365-4dc6-b4c6-c09fc1da2746 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:04:05.078219 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7b0b3d1-3e14-4d8f-9a0a-5e57456770ae None None] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:04:05.078508 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7b0b3d1-3e14-4d8f-9a0a-5e57456770ae None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:04:05.079828 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7b0b3d1-3e14-4d8f-9a0a-5e57456770ae None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:04:05.101550 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.auth.core [None req-f7b0b3d1-3e14-4d8f-9a0a-5e57456770ae None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81578) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:04:05.104113 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-f7b0b3d1-3e14-4d8f-9a0a-5e57456770ae None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:04:05.118265 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1129/2263] 10.222.0.27 () {58 vars in 958 bytes} [Tue Aug 30 02:04:05 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 38 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 30 02:04:05.120470 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7b0b3d1-3e14-4d8f-9a0a-5e57456770ae None None] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:04:05.131542 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7b0b3d1-3e14-4d8f-9a0a-5e57456770ae None None] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:04:05.131816 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db3bf5ef-9365-4dc6-b4c6-c09fc1da2746 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:04:05.137651 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cb53e28-cfa0-4ff2-be8a-340111f5f449 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:04:05.138444 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cb53e28-cfa0-4ff2-be8a-340111f5f449 admin admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:04:05.138731 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cb53e28-cfa0-4ff2-be8a-340111f5f449 admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:04:05.138870 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de2e795e-ce49-4533-9673-e57d6327878a service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f3ea1580c56f4814a92bbb2acf542905', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4562f3951fbe465280073e60a11e44e2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:04:05.139006 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cb53e28-cfa0-4ff2-be8a-340111f5f449 admin admin] PATH_INFO: `/v3/domains` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:04:05.139655 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8cb53e28-cfa0-4ff2-be8a-340111f5f449 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81578) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 02:04:05.139978 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8cb53e28-cfa0-4ff2-be8a-340111f5f449 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:04:05.140079 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de2e795e-ce49-4533-9673-e57d6327878a service neutron] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:04:05.140482 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de2e795e-ce49-4533-9673-e57d6327878a service neutron] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:04:05.141143 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de2e795e-ce49-4533-9673-e57d6327878a service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:04:05.141595 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8cb53e28-cfa0-4ff2-be8a-340111f5f449 admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:04:05.169485 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1130/2264] 10.222.0.27 () {58 vars in 1153 bytes} [Tue Aug 30 02:04:05 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 02:04:05.169671 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de2e795e-ce49-4533-9673-e57d6327878a service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:04:05.169671 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de2e795e-ce49-4533-9673-e57d6327878a service neutron] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:04:05.175858 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cb53e28-cfa0-4ff2-be8a-340111f5f449 admin admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:04:05.180386 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1135/2265] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Aug 30 02:04:05 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 112 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 30 02:04:05.187700 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cb53e28-cfa0-4ff2-be8a-340111f5f449 admin admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:04:05.193693 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b4b75de-e4c4-42a0-a58f-bc66899ed697 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:04:05.193911 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b4b75de-e4c4-42a0-a58f-bc66899ed697 admin admin] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:04:05.194185 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b4b75de-e4c4-42a0-a58f-bc66899ed697 admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:04:05.194526 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b4b75de-e4c4-42a0-a58f-bc66899ed697 admin admin] PATH_INFO: `/v3/projects` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:04:05.195294 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b4b75de-e4c4-42a0-a58f-bc66899ed697 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:04:05.197854 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b4b75de-e4c4-42a0-a58f-bc66899ed697 admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:04:05.319330 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1131/2266] 10.222.0.27 () {60 vars in 1153 bytes} [Tue Aug 30 02:04:05 2022] POST /identity/v3/projects => generated 363 bytes in 144 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 02:04:05.320690 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de2e795e-ce49-4533-9673-e57d6327878a service neutron] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:04:05.343685 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de2e795e-ce49-4533-9673-e57d6327878a service neutron] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:04:05.416154 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0b64640-927e-4545-9d45-528961016c2b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:04:05.416976 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0b64640-927e-4545-9d45-528961016c2b admin admin] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:04:05.417497 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0b64640-927e-4545-9d45-528961016c2b admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:04:05.417968 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0b64640-927e-4545-9d45-528961016c2b admin admin] PATH_INFO: `/v3/users` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:04:05.418615 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0b64640-927e-4545-9d45-528961016c2b admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:04:05.420568 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0b64640-927e-4545-9d45-528961016c2b admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:04:05.584909 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1136/2267] 10.222.0.27 () {60 vars in 1144 bytes} [Tue Aug 30 02:04:05 2022] POST /identity/v3/users => generated 282 bytes in 266 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 02:04:05.590297 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b4b75de-e4c4-42a0-a58f-bc66899ed697 admin admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:04:05.602217 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b4b75de-e4c4-42a0-a58f-bc66899ed697 admin admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:04:05.607651 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b293c72-ebf3-4fa4-b8b4-a9df1a218d04 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:04:05.608237 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b293c72-ebf3-4fa4-b8b4-a9df1a218d04 admin admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:04:05.608540 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b293c72-ebf3-4fa4-b8b4-a9df1a218d04 admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:04:05.608790 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b293c72-ebf3-4fa4-b8b4-a9df1a218d04 admin admin] PATH_INFO: `/v3/roles` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:04:05.609443 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b293c72-ebf3-4fa4-b8b4-a9df1a218d04 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:04:05.611722 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b293c72-ebf3-4fa4-b8b4-a9df1a218d04 admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:04:05.625500 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1132/2268] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Aug 30 02:04:05 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 02:04:05.628547 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0b64640-927e-4545-9d45-528961016c2b admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:04:05.649960 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0b64640-927e-4545-9d45-528961016c2b admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:04:05.651729 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dca2ff62-5eaf-442d-ae42-c8d04ae46a14 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:04:05.652392 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dca2ff62-5eaf-442d-ae42-c8d04ae46a14 admin admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:04:05.652681 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dca2ff62-5eaf-442d-ae42-c8d04ae46a14 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:04:05.652957 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dca2ff62-5eaf-442d-ae42-c8d04ae46a14 admin admin] PATH_INFO: `/v3/roles` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:04:05.653662 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dca2ff62-5eaf-442d-ae42-c8d04ae46a14 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:04:05.655473 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dca2ff62-5eaf-442d-ae42-c8d04ae46a14 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:04:05.667256 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1137/2269] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Aug 30 02:04:05 2022] GET /identity/v3/roles => generated 1420 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 30 02:04:05.671625 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b293c72-ebf3-4fa4-b8b4-a9df1a218d04 admin admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:04:05.682620 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b293c72-ebf3-4fa4-b8b4-a9df1a218d04 admin admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:04:05.684521 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dca2ff62-5eaf-442d-ae42-c8d04ae46a14 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:04:05.686646 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-dca2ff62-5eaf-442d-ae42-c8d04ae46a14 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:04:05.688744 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7eb1b277-346d-4054-bf9c-a4ba0039fb76 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:04:05.691454 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7eb1b277-346d-4054-bf9c-a4ba0039fb76 admin admin] REQUEST_METHOD: `PUT` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:04:05.691454 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7eb1b277-346d-4054-bf9c-a4ba0039fb76 admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:04:05.691454 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7eb1b277-346d-4054-bf9c-a4ba0039fb76 admin admin] PATH_INFO: `/v3/projects/3cd3d5ba273045e5a9fb3b29228887d6/users/4e76bbd5d9234b938d4655dbe6fd6771/roles/1fb81afcb9f245f3b989add94598a063` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:04:05.730634 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7eb1b277-346d-4054-bf9c-a4ba0039fb76 admin admin] RBAC: Authorizing `identity:create_grant(project_id=3cd3d5ba273045e5a9fb3b29228887d6, user_id=4e76bbd5d9234b938d4655dbe6fd6771, role_id=1fb81afcb9f245f3b989add94598a063)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:04:05.734145 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7eb1b277-346d-4054-bf9c-a4ba0039fb76 admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:04:05.739470 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dca2ff62-5eaf-442d-ae42-c8d04ae46a14 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:04:05.746156 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e6d2d28-851d-41ab-86c4-5d197e2bf3ec admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:04:05.746824 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e6d2d28-851d-41ab-86c4-5d197e2bf3ec admin admin] REQUEST_METHOD: `DELETE` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:04:05.750067 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e6d2d28-851d-41ab-86c4-5d197e2bf3ec admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:04:05.750067 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e6d2d28-851d-41ab-86c4-5d197e2bf3ec admin admin] PATH_INFO: `/v3/projects/f29531622fd74dee8e50fa700bf63f04` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:04:05.750067 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e6d2d28-851d-41ab-86c4-5d197e2bf3ec admin admin] RBAC: Authorizing `identity:delete_project(project_id=f29531622fd74dee8e50fa700bf63f04)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:04:05.750424 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e6d2d28-851d-41ab-86c4-5d197e2bf3ec admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:04:05.831521 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=99,keystone:DELETE=1 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:04:05.838091 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1133/2270] 10.222.0.27 () {60 vars in 1483 bytes} [Tue Aug 30 02:04:05 2022] PUT /identity/v3/projects/3cd3d5ba273045e5a9fb3b29228887d6/users/4e76bbd5d9234b938d4655dbe6fd6771/roles/1fb81afcb9f245f3b989add94598a063 => generated 0 bytes in 167 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 02:04:05.839442 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c25d7167-1239-4b3f-a807-b800057e00e3 None None] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:04:05.840739 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c25d7167-1239-4b3f-a807-b800057e00e3 None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:04:05.840739 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c25d7167-1239-4b3f-a807-b800057e00e3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:04:05.867655 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.auth.core [None req-c25d7167-1239-4b3f-a807-b800057e00e3 None None] MFA Rules not processed for user `dbce595e3bec45deb120a3d6fe6d6091`. Rule list: `[]` (Enabled: `True`). {{(pid=81578) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:04:05.868964 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-c25d7167-1239-4b3f-a807-b800057e00e3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:04:05.892028 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-4e6d2d28-851d-41ab-86c4-5d197e2bf3ec admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'f29531622fd74dee8e50fa700bf63f04', 'request_id': 'req-4e6d2d28-851d-41ab-86c4-5d197e2bf3ec'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:04:05.912151 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-4e6d2d28-851d-41ab-86c4-5d197e2bf3ec admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'f29531622fd74dee8e50fa700bf63f04', 'request_id': 'req-4e6d2d28-851d-41ab-86c4-5d197e2bf3ec'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:04:05.921396 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-4e6d2d28-851d-41ab-86c4-5d197e2bf3ec admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'f29531622fd74dee8e50fa700bf63f04', 'request_id': 'req-4e6d2d28-851d-41ab-86c4-5d197e2bf3ec'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:04:05.934143 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1134/2271] 10.222.0.27 () {58 vars in 958 bytes} [Tue Aug 30 02:04:05 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 02:04:05.935427 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d106aa7-d3a2-4ea4-be7c-9c56aa02a70c None None] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:04:05.935709 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d106aa7-d3a2-4ea4-be7c-9c56aa02a70c None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:04:05.935980 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d106aa7-d3a2-4ea4-be7c-9c56aa02a70c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:04:05.965205 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.auth.core [None req-9d106aa7-d3a2-4ea4-be7c-9c56aa02a70c None None] MFA Rules not processed for user `4e76bbd5d9234b938d4655dbe6fd6771`. Rule list: `[]` (Enabled: `True`). {{(pid=81578) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:04:05.966856 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-9d106aa7-d3a2-4ea4-be7c-9c56aa02a70c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:04:06.034186 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1135/2272] 10.222.0.27 () {58 vars in 958 bytes} [Tue Aug 30 02:04:05 2022] POST /identity/v3/auth/tokens => generated 3240 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 02:04:06.046315 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d106aa7-d3a2-4ea4-be7c-9c56aa02a70c None None] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:04:06.048948 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-9d106aa7-d3a2-4ea4-be7c-9c56aa02a70c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:04:06.097187 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-4e6d2d28-851d-41ab-86c4-5d197e2bf3ec admin admin] The token cache is being invalidate because project f29531622fd74dee8e50fa700bf63f04 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81577) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 30 02:04:06.101392 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-4e6d2d28-851d-41ab-86c4-5d197e2bf3ec admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:04:06.101392 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-4e6d2d28-851d-41ab-86c4-5d197e2bf3ec admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:04:06.101392 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1138/2273] 10.222.0.27 () {58 vars in 1233 bytes} [Tue Aug 30 02:04:05 2022] DELETE /identity/v3/projects/f29531622fd74dee8e50fa700bf63f04 => generated 0 bytes in 416 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 02:04:06.105175 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e6d2d28-851d-41ab-86c4-5d197e2bf3ec admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:04:06.107413 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-4e6d2d28-851d-41ab-86c4-5d197e2bf3ec admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:04:06.122449 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d106aa7-d3a2-4ea4-be7c-9c56aa02a70c None None] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:04:06.127692 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4f215c6-7639-4841-acd2-7432fa38abff service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ab936b3767a8417aa80c8c8f1b84b667', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4562f3951fbe465280073e60a11e44e2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:04:06.128650 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4f215c6-7639-4841-acd2-7432fa38abff service cinder] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:04:06.128949 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4f215c6-7639-4841-acd2-7432fa38abff service cinder] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:04:06.129487 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4f215c6-7639-4841-acd2-7432fa38abff service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:04:06.131600 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-b4f215c6-7639-4841-acd2-7432fa38abff service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:04:06.146731 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4f215c6-7639-4841-acd2-7432fa38abff service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:04:06.149036 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4f215c6-7639-4841-acd2-7432fa38abff service cinder] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:04:06.160853 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1136/2274] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Aug 30 02:04:06 2022] GET /identity/v3/auth/tokens => generated 3240 bytes in 115 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 30 02:04:06.187342 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e6d2d28-851d-41ab-86c4-5d197e2bf3ec admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:04:06.193060 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09d54802-c241-4a72-8160-848315607404 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:04:06.201321 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09d54802-c241-4a72-8160-848315607404 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:04:06.201321 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09d54802-c241-4a72-8160-848315607404 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:04:06.201321 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09d54802-c241-4a72-8160-848315607404 admin admin] PATH_INFO: `/v3/users/57e8680db0db4557ad5949849c10e681` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:04:06.201321 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09d54802-c241-4a72-8160-848315607404 admin admin] RBAC: Authorizing `identity:delete_user(user_id=57e8680db0db4557ad5949849c10e681)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:04:06.201321 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09d54802-c241-4a72-8160-848315607404 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:04:06.335362 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:04:06.446214 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-09d54802-c241-4a72-8160-848315607404 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '57e8680db0db4557ad5949849c10e681'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:04:06.446992 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-09d54802-c241-4a72-8160-848315607404 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '57e8680db0db4557ad5949849c10e681'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:04:06.447623 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-09d54802-c241-4a72-8160-848315607404 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '57e8680db0db4557ad5949849c10e681'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:04:06.459397 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-09d54802-c241-4a72-8160-848315607404 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '57e8680db0db4557ad5949849c10e681'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:04:06.467116 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-09d54802-c241-4a72-8160-848315607404 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '57e8680db0db4557ad5949849c10e681'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:04:06.596842 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1139/2275] 10.222.0.27 () {58 vars in 1224 bytes} [Tue Aug 30 02:04:06 2022] DELETE /identity/v3/users/57e8680db0db4557ad5949849c10e681 => generated 0 bytes in 492 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 02:04:06.948315 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=82,keystone:DELETE=16,keystone:INSERT=2 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:04:07.170437 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4f215c6-7639-4841-acd2-7432fa38abff service cinder] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:04:07.190337 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4f215c6-7639-4841-acd2-7432fa38abff service cinder] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:04:07.336201 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0dff440f-acb2-47c1-97f7-8b4af55d9751 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:04:07.336606 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dff440f-acb2-47c1-97f7-8b4af55d9751 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:04:07.336961 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dff440f-acb2-47c1-97f7-8b4af55d9751 admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:04:07.337410 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dff440f-acb2-47c1-97f7-8b4af55d9751 admin admin] PATH_INFO: `/v3/projects/b6275eb14bac45e39bc26836dd7812ba` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:04:07.338602 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0dff440f-acb2-47c1-97f7-8b4af55d9751 admin admin] RBAC: Authorizing `identity:delete_project(project_id=b6275eb14bac45e39bc26836dd7812ba)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:04:07.340514 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0dff440f-acb2-47c1-97f7-8b4af55d9751 admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:04:07.392051 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:04:07.537628 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-0dff440f-acb2-47c1-97f7-8b4af55d9751 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'b6275eb14bac45e39bc26836dd7812ba', 'request_id': 'req-0dff440f-acb2-47c1-97f7-8b4af55d9751'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:04:07.544373 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-0dff440f-acb2-47c1-97f7-8b4af55d9751 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'b6275eb14bac45e39bc26836dd7812ba', 'request_id': 'req-0dff440f-acb2-47c1-97f7-8b4af55d9751'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:04:07.566048 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-0dff440f-acb2-47c1-97f7-8b4af55d9751 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'b6275eb14bac45e39bc26836dd7812ba', 'request_id': 'req-0dff440f-acb2-47c1-97f7-8b4af55d9751'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:04:07.700937 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-0dff440f-acb2-47c1-97f7-8b4af55d9751 admin admin] The token cache is being invalidate because project b6275eb14bac45e39bc26836dd7812ba was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81578) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 30 02:04:07.701239 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-0dff440f-acb2-47c1-97f7-8b4af55d9751 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:04:07.705786 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-0dff440f-acb2-47c1-97f7-8b4af55d9751 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:04:07.705786 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1137/2276] 10.222.0.27 () {58 vars in 1233 bytes} [Tue Aug 30 02:04:07 2022] DELETE /identity/v3/projects/b6275eb14bac45e39bc26836dd7812ba => generated 0 bytes in 534 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 02:04:07.711880 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dbae230-b080-460d-a545-03a4b5299756 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:04:07.712366 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dbae230-b080-460d-a545-03a4b5299756 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:04:07.712456 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dbae230-b080-460d-a545-03a4b5299756 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:04:07.735162 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-2dbae230-b080-460d-a545-03a4b5299756 None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:04:07.736399 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-2dbae230-b080-460d-a545-03a4b5299756 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:04:07.806057 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1140/2277] 10.222.0.27 () {58 vars in 958 bytes} [Tue Aug 30 02:04:07 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 30 02:04:07.809281 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-369dee6a-3314-4e3a-b09e-b5660835667b None None] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:04:07.810257 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-369dee6a-3314-4e3a-b09e-b5660835667b None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:04:07.810257 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-369dee6a-3314-4e3a-b09e-b5660835667b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:04:07.848686 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.auth.core [None req-369dee6a-3314-4e3a-b09e-b5660835667b None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81578) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:04:07.851538 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-369dee6a-3314-4e3a-b09e-b5660835667b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:04:07.862272 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1138/2278] 10.222.0.27 () {58 vars in 958 bytes} [Tue Aug 30 02:04:07 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 54 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 02:04:07.867756 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2dbae230-b080-460d-a545-03a4b5299756 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:04:07.892755 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2dbae230-b080-460d-a545-03a4b5299756 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:04:07.894690 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b04bc3c-daaf-41f4-9479-30b9c7b78f1c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:04:07.895626 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b04bc3c-daaf-41f4-9479-30b9c7b78f1c admin admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:04:07.895626 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b04bc3c-daaf-41f4-9479-30b9c7b78f1c admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:04:07.895825 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b04bc3c-daaf-41f4-9479-30b9c7b78f1c admin admin] PATH_INFO: `/v3/domains` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:04:07.896462 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b04bc3c-daaf-41f4-9479-30b9c7b78f1c admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 02:04:07.896776 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b04bc3c-daaf-41f4-9479-30b9c7b78f1c admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:04:07.898927 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b04bc3c-daaf-41f4-9479-30b9c7b78f1c admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:04:07.913134 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1141/2279] 10.222.0.27 () {58 vars in 1153 bytes} [Tue Aug 30 02:04:07 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 44 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 02:04:07.919200 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-369dee6a-3314-4e3a-b09e-b5660835667b None None] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:04:07.929379 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-369dee6a-3314-4e3a-b09e-b5660835667b None None] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:04:07.936698 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d669ecd-a4cd-4bb0-b1a7-4410b05cd6a9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:04:07.937423 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d669ecd-a4cd-4bb0-b1a7-4410b05cd6a9 admin admin] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:04:07.939826 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d669ecd-a4cd-4bb0-b1a7-4410b05cd6a9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:04:07.940088 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d669ecd-a4cd-4bb0-b1a7-4410b05cd6a9 admin admin] PATH_INFO: `/v3/projects` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:04:07.940808 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d669ecd-a4cd-4bb0-b1a7-4410b05cd6a9 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:04:07.943460 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d669ecd-a4cd-4bb0-b1a7-4410b05cd6a9 admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:04:08.125871 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=91,keystone:DELETE=8,keystone:INSERT=1 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:04:08.136655 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1139/2280] 10.222.0.27 () {60 vars in 1153 bytes} [Tue Aug 30 02:04:07 2022] POST /identity/v3/projects => generated 383 bytes in 218 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 02:04:08.141791 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b04bc3c-daaf-41f4-9479-30b9c7b78f1c admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:04:08.160278 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b04bc3c-daaf-41f4-9479-30b9c7b78f1c admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:04:08.219353 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83f205da-cc89-48f8-ba2c-1467a318b09d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:04:08.220022 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83f205da-cc89-48f8-ba2c-1467a318b09d admin admin] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:04:08.220309 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83f205da-cc89-48f8-ba2c-1467a318b09d admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:04:08.220611 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83f205da-cc89-48f8-ba2c-1467a318b09d admin admin] PATH_INFO: `/v3/users` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:04:08.221247 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83f205da-cc89-48f8-ba2c-1467a318b09d admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:04:08.223237 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83f205da-cc89-48f8-ba2c-1467a318b09d admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:04:08.349411 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1142/2281] 10.222.0.27 () {60 vars in 1144 bytes} [Tue Aug 30 02:04:08 2022] POST /identity/v3/users => generated 292 bytes in 208 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 02:04:08.353975 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d669ecd-a4cd-4bb0-b1a7-4410b05cd6a9 admin admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:04:08.365642 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d669ecd-a4cd-4bb0-b1a7-4410b05cd6a9 admin admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:04:08.370847 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fdeff50-6dfc-4be1-a6f9-6cc4e03e12d3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:04:08.371586 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fdeff50-6dfc-4be1-a6f9-6cc4e03e12d3 admin admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:04:08.372110 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fdeff50-6dfc-4be1-a6f9-6cc4e03e12d3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:04:08.372475 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fdeff50-6dfc-4be1-a6f9-6cc4e03e12d3 admin admin] PATH_INFO: `/v3/roles` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:04:08.373349 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4fdeff50-6dfc-4be1-a6f9-6cc4e03e12d3 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:04:08.375144 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4fdeff50-6dfc-4be1-a6f9-6cc4e03e12d3 admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:04:08.386552 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1140/2282] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Aug 30 02:04:08 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 02:04:08.400634 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83f205da-cc89-48f8-ba2c-1467a318b09d admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:04:08.401777 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83f205da-cc89-48f8-ba2c-1467a318b09d admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:04:08.411245 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0783fa6-bb0f-4943-9525-3a2eb9842a17 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:04:08.412000 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0783fa6-bb0f-4943-9525-3a2eb9842a17 admin admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:04:08.412608 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0783fa6-bb0f-4943-9525-3a2eb9842a17 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:04:08.413708 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0783fa6-bb0f-4943-9525-3a2eb9842a17 admin admin] PATH_INFO: `/v3/roles` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:04:08.414662 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0783fa6-bb0f-4943-9525-3a2eb9842a17 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:04:08.416315 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0783fa6-bb0f-4943-9525-3a2eb9842a17 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:04:08.427445 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1143/2283] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Aug 30 02:04:08 2022] GET /identity/v3/roles => generated 1420 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 02:04:08.431899 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fdeff50-6dfc-4be1-a6f9-6cc4e03e12d3 admin admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:04:08.441794 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fdeff50-6dfc-4be1-a6f9-6cc4e03e12d3 admin admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:04:08.450693 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-920ec3f3-db31-4be2-9687-c3dbfbda9839 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:04:08.450693 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-920ec3f3-db31-4be2-9687-c3dbfbda9839 admin admin] REQUEST_METHOD: `PUT` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:04:08.450693 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-920ec3f3-db31-4be2-9687-c3dbfbda9839 admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:04:08.450693 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-920ec3f3-db31-4be2-9687-c3dbfbda9839 admin admin] PATH_INFO: `/v3/projects/8a1d09b1710d4aa98b9728b133489088/users/88a65df6727b4becaef11e474b421206/roles/1fb81afcb9f245f3b989add94598a063` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:04:08.464851 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-920ec3f3-db31-4be2-9687-c3dbfbda9839 admin admin] RBAC: Authorizing `identity:create_grant(project_id=8a1d09b1710d4aa98b9728b133489088, user_id=88a65df6727b4becaef11e474b421206, role_id=1fb81afcb9f245f3b989add94598a063)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:04:08.467835 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-920ec3f3-db31-4be2-9687-c3dbfbda9839 admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:04:08.587712 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1141/2284] 10.222.0.27 () {60 vars in 1483 bytes} [Tue Aug 30 02:04:08 2022] PUT /identity/v3/projects/8a1d09b1710d4aa98b9728b133489088/users/88a65df6727b4becaef11e474b421206/roles/1fb81afcb9f245f3b989add94598a063 => generated 0 bytes in 156 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 02:04:08.600768 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06793634-7706-4c66-a82d-2e92e2f6fcd7 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:04:08.601124 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06793634-7706-4c66-a82d-2e92e2f6fcd7 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:04:08.601505 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06793634-7706-4c66-a82d-2e92e2f6fcd7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:04:08.629351 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-06793634-7706-4c66-a82d-2e92e2f6fcd7 None None] MFA Rules not processed for user `88a65df6727b4becaef11e474b421206`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:04:08.630637 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-06793634-7706-4c66-a82d-2e92e2f6fcd7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:04:08.693712 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1144/2285] 10.222.0.27 () {58 vars in 958 bytes} [Tue Aug 30 02:04:08 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 02:04:08.705132 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-920ec3f3-db31-4be2-9687-c3dbfbda9839 admin admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:04:08.706936 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-920ec3f3-db31-4be2-9687-c3dbfbda9839 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:04:08.764480 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-920ec3f3-db31-4be2-9687-c3dbfbda9839 admin admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:04:08.768112 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43feb849-e5b2-4da9-9ce1-85c44f3fa118 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ab936b3767a8417aa80c8c8f1b84b667', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4562f3951fbe465280073e60a11e44e2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:04:08.768715 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43feb849-e5b2-4da9-9ce1-85c44f3fa118 service cinder] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:04:08.768994 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43feb849-e5b2-4da9-9ce1-85c44f3fa118 service cinder] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:04:08.769263 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43feb849-e5b2-4da9-9ce1-85c44f3fa118 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:04:08.777431 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43feb849-e5b2-4da9-9ce1-85c44f3fa118 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:04:08.778924 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43feb849-e5b2-4da9-9ce1-85c44f3fa118 service cinder] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:04:08.788445 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1142/2286] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Aug 30 02:04:08 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 84 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 30 02:04:08.809844 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06793634-7706-4c66-a82d-2e92e2f6fcd7 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:04:08.872971 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06793634-7706-4c66-a82d-2e92e2f6fcd7 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:04:08.877630 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ea27ad7-615d-4208-98bc-fd6c02045e54 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:04:08.878259 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ea27ad7-615d-4208-98bc-fd6c02045e54 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:04:08.878536 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ea27ad7-615d-4208-98bc-fd6c02045e54 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:04:08.878806 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ea27ad7-615d-4208-98bc-fd6c02045e54 admin admin] PATH_INFO: `/v3/users/88a65df6727b4becaef11e474b421206` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:04:08.879947 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ea27ad7-615d-4208-98bc-fd6c02045e54 admin admin] RBAC: Authorizing `identity:delete_user(user_id=88a65df6727b4becaef11e474b421206)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:04:08.882008 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ea27ad7-615d-4208-98bc-fd6c02045e54 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:04:09.210961 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-5ea27ad7-615d-4208-98bc-fd6c02045e54 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '88a65df6727b4becaef11e474b421206'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:04:09.211620 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-5ea27ad7-615d-4208-98bc-fd6c02045e54 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '88a65df6727b4becaef11e474b421206'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:04:09.212243 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-5ea27ad7-615d-4208-98bc-fd6c02045e54 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '88a65df6727b4becaef11e474b421206'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:04:09.224680 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-5ea27ad7-615d-4208-98bc-fd6c02045e54 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '88a65df6727b4becaef11e474b421206'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:04:09.232654 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-5ea27ad7-615d-4208-98bc-fd6c02045e54 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '88a65df6727b4becaef11e474b421206'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:04:09.253327 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:04:09.403272 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1145/2287] 10.222.0.27 () {58 vars in 1224 bytes} [Tue Aug 30 02:04:08 2022] DELETE /identity/v3/users/88a65df6727b4becaef11e474b421206 => generated 0 bytes in 594 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 02:04:09.407576 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f541eebf-c937-4049-aef1-70fde7daeba0 None None] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:04:09.408040 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f541eebf-c937-4049-aef1-70fde7daeba0 None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:04:09.408150 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f541eebf-c937-4049-aef1-70fde7daeba0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:04:09.427865 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.auth.core [None req-f541eebf-c937-4049-aef1-70fde7daeba0 None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81578) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:04:09.429152 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-f541eebf-c937-4049-aef1-70fde7daeba0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:04:09.495833 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1143/2288] 10.222.0.27 () {58 vars in 958 bytes} [Tue Aug 30 02:04:09 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 02:04:09.505327 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ea27ad7-615d-4208-98bc-fd6c02045e54 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:04:09.506893 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-5ea27ad7-615d-4208-98bc-fd6c02045e54 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:04:09.564071 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ea27ad7-615d-4208-98bc-fd6c02045e54 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:04:09.567634 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61919239-ec8a-4673-a768-45e6bde185bb service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f3ea1580c56f4814a92bbb2acf542905', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4562f3951fbe465280073e60a11e44e2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:04:09.568249 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61919239-ec8a-4673-a768-45e6bde185bb service neutron] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:04:09.568508 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61919239-ec8a-4673-a768-45e6bde185bb service neutron] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:04:09.568894 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61919239-ec8a-4673-a768-45e6bde185bb service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:04:09.577771 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61919239-ec8a-4673-a768-45e6bde185bb service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:04:09.579190 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61919239-ec8a-4673-a768-45e6bde185bb service neutron] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:04:09.588899 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1146/2289] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Aug 30 02:04:09 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 02:04:10.032939 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f541eebf-c937-4049-aef1-70fde7daeba0 None None] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:04:10.034493 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-f541eebf-c937-4049-aef1-70fde7daeba0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:04:10.047830 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f541eebf-c937-4049-aef1-70fde7daeba0 None None] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:04:10.052602 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a168ab13-61ca-4d94-84ed-3b201df5ee14 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:04:10.054249 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a168ab13-61ca-4d94-84ed-3b201df5ee14 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:04:10.054698 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a168ab13-61ca-4d94-84ed-3b201df5ee14 admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:04:10.055062 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a168ab13-61ca-4d94-84ed-3b201df5ee14 admin admin] PATH_INFO: `/v3/projects/8a1d09b1710d4aa98b9728b133489088` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:04:10.056460 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a168ab13-61ca-4d94-84ed-3b201df5ee14 admin admin] RBAC: Authorizing `identity:delete_project(project_id=8a1d09b1710d4aa98b9728b133489088)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:04:10.058975 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a168ab13-61ca-4d94-84ed-3b201df5ee14 admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:04:10.244010 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=95,keystone:DELETE=2,keystone:INSERT=3 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:04:10.247103 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-a168ab13-61ca-4d94-84ed-3b201df5ee14 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '8a1d09b1710d4aa98b9728b133489088', 'request_id': 'req-a168ab13-61ca-4d94-84ed-3b201df5ee14'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:04:10.252958 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-a168ab13-61ca-4d94-84ed-3b201df5ee14 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '8a1d09b1710d4aa98b9728b133489088', 'request_id': 'req-a168ab13-61ca-4d94-84ed-3b201df5ee14'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:04:10.259641 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-a168ab13-61ca-4d94-84ed-3b201df5ee14 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '8a1d09b1710d4aa98b9728b133489088', 'request_id': 'req-a168ab13-61ca-4d94-84ed-3b201df5ee14'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:04:10.391777 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-a168ab13-61ca-4d94-84ed-3b201df5ee14 admin admin] The token cache is being invalidate because project 8a1d09b1710d4aa98b9728b133489088 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81578) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 30 02:04:10.392072 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-a168ab13-61ca-4d94-84ed-3b201df5ee14 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:04:10.392725 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-a168ab13-61ca-4d94-84ed-3b201df5ee14 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:04:10.394361 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1144/2290] 10.222.0.27 () {58 vars in 1233 bytes} [Tue Aug 30 02:04:10 2022] DELETE /identity/v3/projects/8a1d09b1710d4aa98b9728b133489088 => generated 0 bytes in 362 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 02:04:10.400215 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0d6b59e-2a68-4b29-9bc4-6e6e751b20ed None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:04:10.400729 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0d6b59e-2a68-4b29-9bc4-6e6e751b20ed None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:04:10.400824 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0d6b59e-2a68-4b29-9bc4-6e6e751b20ed None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:04:10.421008 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-e0d6b59e-2a68-4b29-9bc4-6e6e751b20ed None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:04:10.422727 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-e0d6b59e-2a68-4b29-9bc4-6e6e751b20ed None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:04:10.483820 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1147/2291] 10.222.0.27 () {58 vars in 958 bytes} [Tue Aug 30 02:04:10 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 30 02:04:10.492076 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e841979-95cc-40c8-b4ba-1296b18151bc None None] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:04:10.492524 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e841979-95cc-40c8-b4ba-1296b18151bc None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:04:10.492806 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e841979-95cc-40c8-b4ba-1296b18151bc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:04:10.515277 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.auth.core [None req-1e841979-95cc-40c8-b4ba-1296b18151bc None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81578) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:04:10.516547 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-1e841979-95cc-40c8-b4ba-1296b18151bc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:04:10.527627 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1145/2292] 10.222.0.27 () {58 vars in 958 bytes} [Tue Aug 30 02:04:10 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 37 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 02:04:10.529995 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0d6b59e-2a68-4b29-9bc4-6e6e751b20ed None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:04:10.540173 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0d6b59e-2a68-4b29-9bc4-6e6e751b20ed None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:04:10.544907 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f10bc559-4a56-4bb5-8994-a2379a49acac admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:04:10.545540 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f10bc559-4a56-4bb5-8994-a2379a49acac admin admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:04:10.545817 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f10bc559-4a56-4bb5-8994-a2379a49acac admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:04:10.546294 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f10bc559-4a56-4bb5-8994-a2379a49acac admin admin] PATH_INFO: `/v3/domains` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:04:10.546948 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f10bc559-4a56-4bb5-8994-a2379a49acac admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 02:04:10.547246 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f10bc559-4a56-4bb5-8994-a2379a49acac admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:04:10.548826 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f10bc559-4a56-4bb5-8994-a2379a49acac admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:04:10.561399 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1148/2293] 10.222.0.27 () {58 vars in 1153 bytes} [Tue Aug 30 02:04:10 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 02:04:10.565638 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e841979-95cc-40c8-b4ba-1296b18151bc None None] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:04:10.574960 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e841979-95cc-40c8-b4ba-1296b18151bc None None] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:04:10.579657 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09045c30-a0f6-44fc-a280-5d188f7f33e9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:04:10.580261 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09045c30-a0f6-44fc-a280-5d188f7f33e9 admin admin] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:04:10.580541 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09045c30-a0f6-44fc-a280-5d188f7f33e9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:04:10.580811 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09045c30-a0f6-44fc-a280-5d188f7f33e9 admin admin] PATH_INFO: `/v3/projects` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:04:10.581447 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09045c30-a0f6-44fc-a280-5d188f7f33e9 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:04:10.583369 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09045c30-a0f6-44fc-a280-5d188f7f33e9 admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:04:10.719736 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1146/2294] 10.222.0.27 () {60 vars in 1153 bytes} [Tue Aug 30 02:04:10 2022] POST /identity/v3/projects => generated 375 bytes in 155 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 30 02:04:10.724069 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f10bc559-4a56-4bb5-8994-a2379a49acac admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:04:10.734562 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f10bc559-4a56-4bb5-8994-a2379a49acac admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:04:10.792367 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a7e00c8-fd22-4c24-9490-46b336af25a9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:04:10.792899 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a7e00c8-fd22-4c24-9490-46b336af25a9 admin admin] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:04:10.793183 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a7e00c8-fd22-4c24-9490-46b336af25a9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:04:10.793473 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a7e00c8-fd22-4c24-9490-46b336af25a9 admin admin] PATH_INFO: `/v3/users` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:04:10.794131 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a7e00c8-fd22-4c24-9490-46b336af25a9 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:04:10.796054 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a7e00c8-fd22-4c24-9490-46b336af25a9 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:04:10.906789 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1149/2295] 10.222.0.27 () {60 vars in 1144 bytes} [Tue Aug 30 02:04:10 2022] POST /identity/v3/users => generated 288 bytes in 183 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 30 02:04:10.911280 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09045c30-a0f6-44fc-a280-5d188f7f33e9 admin admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:04:10.921477 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09045c30-a0f6-44fc-a280-5d188f7f33e9 admin admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:04:10.926472 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cba76d6-11ef-4f3f-8e43-28f558adc2d3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:04:10.927141 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cba76d6-11ef-4f3f-8e43-28f558adc2d3 admin admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:04:10.927437 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cba76d6-11ef-4f3f-8e43-28f558adc2d3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:04:10.927711 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cba76d6-11ef-4f3f-8e43-28f558adc2d3 admin admin] PATH_INFO: `/v3/roles` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:04:10.928390 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cba76d6-11ef-4f3f-8e43-28f558adc2d3 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:04:10.930057 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cba76d6-11ef-4f3f-8e43-28f558adc2d3 admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:04:10.940009 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1147/2296] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Aug 30 02:04:10 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 02:04:10.944737 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a7e00c8-fd22-4c24-9490-46b336af25a9 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:04:10.954814 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a7e00c8-fd22-4c24-9490-46b336af25a9 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:04:10.960453 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a24e4ba-0d58-4b04-9ebc-11d5a2cb738a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:04:10.961074 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a24e4ba-0d58-4b04-9ebc-11d5a2cb738a admin admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:04:10.961354 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a24e4ba-0d58-4b04-9ebc-11d5a2cb738a admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:04:10.961664 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a24e4ba-0d58-4b04-9ebc-11d5a2cb738a admin admin] PATH_INFO: `/v3/roles` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:04:10.962309 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a24e4ba-0d58-4b04-9ebc-11d5a2cb738a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:04:10.963909 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a24e4ba-0d58-4b04-9ebc-11d5a2cb738a admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:04:10.976698 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1150/2297] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Aug 30 02:04:10 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 02:04:10.981228 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cba76d6-11ef-4f3f-8e43-28f558adc2d3 admin admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:04:10.993512 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cba76d6-11ef-4f3f-8e43-28f558adc2d3 admin admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:04:10.998421 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdb3fe8e-909f-4644-8706-a6410dcd852c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:04:10.999081 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdb3fe8e-909f-4644-8706-a6410dcd852c admin admin] REQUEST_METHOD: `PUT` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:04:10.999363 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdb3fe8e-909f-4644-8706-a6410dcd852c admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:04:10.999640 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdb3fe8e-909f-4644-8706-a6410dcd852c admin admin] PATH_INFO: `/v3/projects/fb4e4792c04e48ab9f1fc222832612be/users/7b8810c558074c4cb91d414cae952fdc/roles/1fb81afcb9f245f3b989add94598a063` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:04:11.015272 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bdb3fe8e-909f-4644-8706-a6410dcd852c admin admin] RBAC: Authorizing `identity:create_grant(project_id=fb4e4792c04e48ab9f1fc222832612be, user_id=7b8810c558074c4cb91d414cae952fdc, role_id=1fb81afcb9f245f3b989add94598a063)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:04:11.019916 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bdb3fe8e-909f-4644-8706-a6410dcd852c admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:04:11.065959 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:04:11.100378 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1148/2298] 10.222.0.27 () {60 vars in 1483 bytes} [Tue Aug 30 02:04:10 2022] PUT /identity/v3/projects/fb4e4792c04e48ab9f1fc222832612be/users/7b8810c558074c4cb91d414cae952fdc/roles/1fb81afcb9f245f3b989add94598a063 => generated 0 bytes in 120 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 02:04:11.111748 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da05a043-c074-435b-89bd-79f581be245b None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:04:11.112018 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da05a043-c074-435b-89bd-79f581be245b None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:04:11.112461 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da05a043-c074-435b-89bd-79f581be245b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:04:11.146197 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-da05a043-c074-435b-89bd-79f581be245b None None] MFA Rules not processed for user `7b8810c558074c4cb91d414cae952fdc`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:04:11.146197 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-da05a043-c074-435b-89bd-79f581be245b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:04:11.236977 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1151/2299] 10.222.0.27 () {58 vars in 958 bytes} [Tue Aug 30 02:04:11 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 127 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 02:04:11.248175 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdb3fe8e-909f-4644-8706-a6410dcd852c admin admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:04:11.249909 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-bdb3fe8e-909f-4644-8706-a6410dcd852c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:04:11.309660 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdb3fe8e-909f-4644-8706-a6410dcd852c admin admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:04:11.317925 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75551cc5-8d08-4b16-be0b-e0bb469285bf service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ab936b3767a8417aa80c8c8f1b84b667', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4562f3951fbe465280073e60a11e44e2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:04:11.317925 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75551cc5-8d08-4b16-be0b-e0bb469285bf service cinder] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:04:11.317925 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75551cc5-8d08-4b16-be0b-e0bb469285bf service cinder] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:04:11.317925 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75551cc5-8d08-4b16-be0b-e0bb469285bf service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:04:11.324968 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75551cc5-8d08-4b16-be0b-e0bb469285bf service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:04:11.326521 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75551cc5-8d08-4b16-be0b-e0bb469285bf service cinder] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:04:11.338780 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1149/2300] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Aug 30 02:04:11 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 91 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 02:04:12.065926 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:04:12.286550 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:04:13.551281 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=87,keystone:DELETE=12,keystone:INSERT=1 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:04:14.185595 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=86,keystone:DELETE=10,keystone:INSERT=4 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:04:14.198306 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da05a043-c074-435b-89bd-79f581be245b None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:04:14.199801 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-da05a043-c074-435b-89bd-79f581be245b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:04:14.270963 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da05a043-c074-435b-89bd-79f581be245b None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:04:14.275598 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3660ee81-a4d8-4227-90ae-bf3ad38aa413 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:04:14.276234 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3660ee81-a4d8-4227-90ae-bf3ad38aa413 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:04:14.276521 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3660ee81-a4d8-4227-90ae-bf3ad38aa413 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:04:14.276793 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3660ee81-a4d8-4227-90ae-bf3ad38aa413 admin admin] PATH_INFO: `/v3/users/de5de5bbd1004b2d83160ef0268b1ac2` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:04:14.278040 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3660ee81-a4d8-4227-90ae-bf3ad38aa413 admin admin] RBAC: Authorizing `identity:delete_user(user_id=de5de5bbd1004b2d83160ef0268b1ac2)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:04:14.279967 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3660ee81-a4d8-4227-90ae-bf3ad38aa413 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:04:14.519164 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-3660ee81-a4d8-4227-90ae-bf3ad38aa413 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'de5de5bbd1004b2d83160ef0268b1ac2'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:04:14.519813 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-3660ee81-a4d8-4227-90ae-bf3ad38aa413 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'de5de5bbd1004b2d83160ef0268b1ac2'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:04:14.520421 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-3660ee81-a4d8-4227-90ae-bf3ad38aa413 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'de5de5bbd1004b2d83160ef0268b1ac2'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:04:14.532716 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-3660ee81-a4d8-4227-90ae-bf3ad38aa413 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'de5de5bbd1004b2d83160ef0268b1ac2'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:04:14.540331 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-3660ee81-a4d8-4227-90ae-bf3ad38aa413 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'de5de5bbd1004b2d83160ef0268b1ac2'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:04:14.683459 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1152/2301] 10.222.0.27 () {58 vars in 1224 bytes} [Tue Aug 30 02:04:14 2022] DELETE /identity/v3/users/de5de5bbd1004b2d83160ef0268b1ac2 => generated 0 bytes in 486 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 02:04:14.948719 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75551cc5-8d08-4b16-be0b-e0bb469285bf service cinder] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:04:14.960539 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75551cc5-8d08-4b16-be0b-e0bb469285bf service cinder] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:04:15.019447 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d82d9db3-290c-42c6-814b-a31ef16ef732 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:04:15.020157 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d82d9db3-290c-42c6-814b-a31ef16ef732 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:04:15.020439 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d82d9db3-290c-42c6-814b-a31ef16ef732 admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:04:15.020712 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d82d9db3-290c-42c6-814b-a31ef16ef732 admin admin] PATH_INFO: `/v3/projects/f91bad8afa8a44d4ae6a4d31d8f5bfdf` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:04:15.027941 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d82d9db3-290c-42c6-814b-a31ef16ef732 admin admin] RBAC: Authorizing `identity:delete_project(project_id=f91bad8afa8a44d4ae6a4d31d8f5bfdf)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:04:15.032128 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d82d9db3-290c-42c6-814b-a31ef16ef732 admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:04:15.093928 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=94,keystone:DELETE=2,keystone:INSERT=4 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:04:15.181004 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-d82d9db3-290c-42c6-814b-a31ef16ef732 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'f91bad8afa8a44d4ae6a4d31d8f5bfdf', 'request_id': 'req-d82d9db3-290c-42c6-814b-a31ef16ef732'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:04:15.186602 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-d82d9db3-290c-42c6-814b-a31ef16ef732 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'f91bad8afa8a44d4ae6a4d31d8f5bfdf', 'request_id': 'req-d82d9db3-290c-42c6-814b-a31ef16ef732'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:04:15.192910 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-d82d9db3-290c-42c6-814b-a31ef16ef732 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'f91bad8afa8a44d4ae6a4d31d8f5bfdf', 'request_id': 'req-d82d9db3-290c-42c6-814b-a31ef16ef732'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:04:15.316108 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-d82d9db3-290c-42c6-814b-a31ef16ef732 admin admin] The token cache is being invalidate because project f91bad8afa8a44d4ae6a4d31d8f5bfdf was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81578) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 30 02:04:15.316578 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-d82d9db3-290c-42c6-814b-a31ef16ef732 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:04:15.317091 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-d82d9db3-290c-42c6-814b-a31ef16ef732 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:04:15.319384 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1150/2302] 10.222.0.27 () {58 vars in 1233 bytes} [Tue Aug 30 02:04:14 2022] DELETE /identity/v3/projects/f91bad8afa8a44d4ae6a4d31d8f5bfdf => generated 0 bytes in 371 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 02:04:15.323530 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3660ee81-a4d8-4227-90ae-bf3ad38aa413 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:04:15.325109 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-3660ee81-a4d8-4227-90ae-bf3ad38aa413 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:04:15.340062 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3660ee81-a4d8-4227-90ae-bf3ad38aa413 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:04:15.397910 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5739e460-a3cb-4f8c-b75b-49d3cbfac3da admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:04:15.398549 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5739e460-a3cb-4f8c-b75b-49d3cbfac3da admin admin] REQUEST_METHOD: `DELETE` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:04:15.398854 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5739e460-a3cb-4f8c-b75b-49d3cbfac3da admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:04:15.399128 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5739e460-a3cb-4f8c-b75b-49d3cbfac3da admin admin] PATH_INFO: `/v3/users/f865d9c0901849269956da4aad9a7d88` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:04:15.400258 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5739e460-a3cb-4f8c-b75b-49d3cbfac3da admin admin] RBAC: Authorizing `identity:delete_user(user_id=f865d9c0901849269956da4aad9a7d88)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:04:15.402222 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5739e460-a3cb-4f8c-b75b-49d3cbfac3da admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:04:15.545851 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:04:15.617677 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-5739e460-a3cb-4f8c-b75b-49d3cbfac3da admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'f865d9c0901849269956da4aad9a7d88'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:04:15.618405 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-5739e460-a3cb-4f8c-b75b-49d3cbfac3da admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'f865d9c0901849269956da4aad9a7d88'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:04:15.619067 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-5739e460-a3cb-4f8c-b75b-49d3cbfac3da admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'f865d9c0901849269956da4aad9a7d88'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:04:15.631829 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-5739e460-a3cb-4f8c-b75b-49d3cbfac3da admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'f865d9c0901849269956da4aad9a7d88'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:04:15.638992 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-5739e460-a3cb-4f8c-b75b-49d3cbfac3da admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'f865d9c0901849269956da4aad9a7d88'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:04:15.750802 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1153/2303] 10.222.0.27 () {58 vars in 1224 bytes} [Tue Aug 30 02:04:15 2022] DELETE /identity/v3/users/f865d9c0901849269956da4aad9a7d88 => generated 0 bytes in 428 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 30 02:04:16.137935 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d82d9db3-290c-42c6-814b-a31ef16ef732 admin admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:04:16.139531 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-d82d9db3-290c-42c6-814b-a31ef16ef732 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:04:16.155721 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d82d9db3-290c-42c6-814b-a31ef16ef732 admin admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:04:16.219573 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6d5d1cf-6635-4e4f-a401-effcc6110b4f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:04:16.219807 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6d5d1cf-6635-4e4f-a401-effcc6110b4f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:04:16.219917 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6d5d1cf-6635-4e4f-a401-effcc6110b4f admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:04:16.220193 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6d5d1cf-6635-4e4f-a401-effcc6110b4f admin admin] PATH_INFO: `/v3/projects/c4ae66af82a54303bcfb9c38a7081688` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:04:16.221313 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6d5d1cf-6635-4e4f-a401-effcc6110b4f admin admin] RBAC: Authorizing `identity:delete_project(project_id=c4ae66af82a54303bcfb9c38a7081688)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:04:16.223320 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6d5d1cf-6635-4e4f-a401-effcc6110b4f admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:04:16.259708 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=95,keystone:DELETE=5 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:04:16.366433 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-e6d5d1cf-6635-4e4f-a401-effcc6110b4f admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'c4ae66af82a54303bcfb9c38a7081688', 'request_id': 'req-e6d5d1cf-6635-4e4f-a401-effcc6110b4f'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:04:16.373675 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-e6d5d1cf-6635-4e4f-a401-effcc6110b4f admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'c4ae66af82a54303bcfb9c38a7081688', 'request_id': 'req-e6d5d1cf-6635-4e4f-a401-effcc6110b4f'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:04:16.382589 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-e6d5d1cf-6635-4e4f-a401-effcc6110b4f admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'c4ae66af82a54303bcfb9c38a7081688', 'request_id': 'req-e6d5d1cf-6635-4e4f-a401-effcc6110b4f'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:04:16.523894 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-e6d5d1cf-6635-4e4f-a401-effcc6110b4f admin admin] The token cache is being invalidate because project c4ae66af82a54303bcfb9c38a7081688 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81578) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 30 02:04:16.524200 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-e6d5d1cf-6635-4e4f-a401-effcc6110b4f admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:04:16.525560 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-e6d5d1cf-6635-4e4f-a401-effcc6110b4f admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:04:16.527128 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1151/2304] 10.222.0.27 () {58 vars in 1233 bytes} [Tue Aug 30 02:04:16 2022] DELETE /identity/v3/projects/c4ae66af82a54303bcfb9c38a7081688 => generated 0 bytes in 390 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 02:04:16.532350 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f61b3621-9de2-469e-9d84-f195481fa966 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:04:16.533235 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f61b3621-9de2-469e-9d84-f195481fa966 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:04:16.533839 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f61b3621-9de2-469e-9d84-f195481fa966 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:04:16.556519 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-f61b3621-9de2-469e-9d84-f195481fa966 None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:04:16.557990 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-f61b3621-9de2-469e-9d84-f195481fa966 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:04:16.604071 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:04:16.620003 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1154/2305] 10.222.0.27 () {58 vars in 958 bytes} [Tue Aug 30 02:04:16 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 02:04:16.627987 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20bfb71f-7728-48b6-838f-e8d6be65d242 None None] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:04:16.628274 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20bfb71f-7728-48b6-838f-e8d6be65d242 None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:04:16.628545 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20bfb71f-7728-48b6-838f-e8d6be65d242 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:04:16.649955 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.auth.core [None req-20bfb71f-7728-48b6-838f-e8d6be65d242 None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81578) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:04:16.651183 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-20bfb71f-7728-48b6-838f-e8d6be65d242 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:04:16.660735 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1152/2306] 10.222.0.27 () {58 vars in 958 bytes} [Tue Aug 30 02:04:16 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 02:04:16.665861 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f61b3621-9de2-469e-9d84-f195481fa966 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:04:16.675365 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f61b3621-9de2-469e-9d84-f195481fa966 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:04:16.680175 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2f6b984-13b3-4c7f-ae4e-6c6fe257834c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:04:16.680772 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2f6b984-13b3-4c7f-ae4e-6c6fe257834c admin admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:04:16.681054 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2f6b984-13b3-4c7f-ae4e-6c6fe257834c admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:04:16.681328 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2f6b984-13b3-4c7f-ae4e-6c6fe257834c admin admin] PATH_INFO: `/v3/domains` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:04:16.682016 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2f6b984-13b3-4c7f-ae4e-6c6fe257834c admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 02:04:16.682348 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2f6b984-13b3-4c7f-ae4e-6c6fe257834c admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:04:16.683946 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2f6b984-13b3-4c7f-ae4e-6c6fe257834c admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:04:16.696265 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1155/2307] 10.222.0.27 () {58 vars in 1153 bytes} [Tue Aug 30 02:04:16 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 02:04:16.700784 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20bfb71f-7728-48b6-838f-e8d6be65d242 None None] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:04:16.711442 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20bfb71f-7728-48b6-838f-e8d6be65d242 None None] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:04:16.717122 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94da6487-7ef2-402c-bc7a-6768e4b620b5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:04:16.717776 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94da6487-7ef2-402c-bc7a-6768e4b620b5 admin admin] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:04:16.718058 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94da6487-7ef2-402c-bc7a-6768e4b620b5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:04:16.718447 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94da6487-7ef2-402c-bc7a-6768e4b620b5 admin admin] PATH_INFO: `/v3/projects` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:04:16.718968 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94da6487-7ef2-402c-bc7a-6768e4b620b5 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:04:16.721035 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94da6487-7ef2-402c-bc7a-6768e4b620b5 admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:04:16.827289 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1153/2308] 10.222.0.27 () {60 vars in 1153 bytes} [Tue Aug 30 02:04:16 2022] POST /identity/v3/projects => generated 383 bytes in 127 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 30 02:04:16.841380 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2f6b984-13b3-4c7f-ae4e-6c6fe257834c admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:04:16.851342 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2f6b984-13b3-4c7f-ae4e-6c6fe257834c admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:04:16.922557 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21a94dd6-ebf1-4a56-a2b5-4e43dad12fdf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:04:16.923170 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21a94dd6-ebf1-4a56-a2b5-4e43dad12fdf admin admin] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:04:16.923447 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21a94dd6-ebf1-4a56-a2b5-4e43dad12fdf admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:04:16.923718 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21a94dd6-ebf1-4a56-a2b5-4e43dad12fdf admin admin] PATH_INFO: `/v3/users` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:04:16.924455 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21a94dd6-ebf1-4a56-a2b5-4e43dad12fdf admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:04:16.930939 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21a94dd6-ebf1-4a56-a2b5-4e43dad12fdf admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:04:17.073501 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1156/2309] 10.222.0.27 () {60 vars in 1144 bytes} [Tue Aug 30 02:04:16 2022] POST /identity/v3/users => generated 292 bytes in 232 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 02:04:17.077443 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94da6487-7ef2-402c-bc7a-6768e4b620b5 admin admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:04:17.089851 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94da6487-7ef2-402c-bc7a-6768e4b620b5 admin admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:04:17.101547 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a398fc2-3176-4691-a2c3-47dfa85485b9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:04:17.101547 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a398fc2-3176-4691-a2c3-47dfa85485b9 admin admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:04:17.101547 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a398fc2-3176-4691-a2c3-47dfa85485b9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:04:17.101547 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a398fc2-3176-4691-a2c3-47dfa85485b9 admin admin] PATH_INFO: `/v3/roles` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:04:17.101890 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a398fc2-3176-4691-a2c3-47dfa85485b9 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:04:17.101890 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a398fc2-3176-4691-a2c3-47dfa85485b9 admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:04:17.116069 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1154/2310] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Aug 30 02:04:17 2022] GET /identity/v3/roles => generated 1420 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 30 02:04:17.118829 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21a94dd6-ebf1-4a56-a2b5-4e43dad12fdf admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:04:17.128987 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21a94dd6-ebf1-4a56-a2b5-4e43dad12fdf admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:04:17.133752 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bb780c1-1431-4656-8577-86a65ef2f28c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:04:17.134375 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bb780c1-1431-4656-8577-86a65ef2f28c admin admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:04:17.134654 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bb780c1-1431-4656-8577-86a65ef2f28c admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:04:17.134925 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bb780c1-1431-4656-8577-86a65ef2f28c admin admin] PATH_INFO: `/v3/roles` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:04:17.135568 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9bb780c1-1431-4656-8577-86a65ef2f28c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:04:17.137246 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9bb780c1-1431-4656-8577-86a65ef2f28c admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:04:17.148926 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1157/2311] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Aug 30 02:04:17 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 02:04:17.153706 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a398fc2-3176-4691-a2c3-47dfa85485b9 admin admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:04:17.167276 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a398fc2-3176-4691-a2c3-47dfa85485b9 admin admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:04:17.172149 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12c1d07f-8047-42a7-9435-570a808f29c9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:04:17.172890 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12c1d07f-8047-42a7-9435-570a808f29c9 admin admin] REQUEST_METHOD: `PUT` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:04:17.173278 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12c1d07f-8047-42a7-9435-570a808f29c9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:04:17.173661 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12c1d07f-8047-42a7-9435-570a808f29c9 admin admin] PATH_INFO: `/v3/projects/a6412607e3e34ed98f036a3f34805345/users/c5e1193d61454052a68a493a8fe896a3/roles/1fb81afcb9f245f3b989add94598a063` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:04:17.194087 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12c1d07f-8047-42a7-9435-570a808f29c9 admin admin] RBAC: Authorizing `identity:create_grant(project_id=a6412607e3e34ed98f036a3f34805345, user_id=c5e1193d61454052a68a493a8fe896a3, role_id=1fb81afcb9f245f3b989add94598a063)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:04:17.197312 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12c1d07f-8047-42a7-9435-570a808f29c9 admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:04:17.337101 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:DELETE=2,keystone:SELECT=93,keystone:INSERT=5 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:04:17.346801 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1155/2312] 10.222.0.27 () {60 vars in 1483 bytes} [Tue Aug 30 02:04:17 2022] PUT /identity/v3/projects/a6412607e3e34ed98f036a3f34805345/users/c5e1193d61454052a68a493a8fe896a3/roles/1fb81afcb9f245f3b989add94598a063 => generated 0 bytes in 193 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 02:04:17.358047 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c7ce8f3-5463-4526-84a1-d4b6b439fded None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:04:17.358967 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c7ce8f3-5463-4526-84a1-d4b6b439fded None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:04:17.359584 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c7ce8f3-5463-4526-84a1-d4b6b439fded None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:04:17.398637 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-2c7ce8f3-5463-4526-84a1-d4b6b439fded None None] MFA Rules not processed for user `c5e1193d61454052a68a493a8fe896a3`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:04:17.430049 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-2c7ce8f3-5463-4526-84a1-d4b6b439fded None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:04:17.619348 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1158/2313] 10.222.0.27 () {58 vars in 958 bytes} [Tue Aug 30 02:04:17 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 262 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 02:04:18.246693 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12c1d07f-8047-42a7-9435-570a808f29c9 admin admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:04:18.251435 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-12c1d07f-8047-42a7-9435-570a808f29c9 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:04:18.311899 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12c1d07f-8047-42a7-9435-570a808f29c9 admin admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:04:18.315605 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ea2faa8-3192-468d-99f5-45bc8c7c6775 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ab936b3767a8417aa80c8c8f1b84b667', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4562f3951fbe465280073e60a11e44e2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:04:18.316214 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ea2faa8-3192-468d-99f5-45bc8c7c6775 service cinder] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:04:18.316493 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ea2faa8-3192-468d-99f5-45bc8c7c6775 service cinder] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:04:18.316763 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ea2faa8-3192-468d-99f5-45bc8c7c6775 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:04:18.328155 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ea2faa8-3192-468d-99f5-45bc8c7c6775 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:04:18.329640 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ea2faa8-3192-468d-99f5-45bc8c7c6775 service cinder] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:04:18.339762 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1156/2314] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Aug 30 02:04:18 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 94 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 02:04:19.276827 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:04:20.722055 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=97,keystone:DELETE=3 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:04:22.762374 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=91,keystone:INSERT=2,keystone:DELETE=7 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:04:22.791149 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:DELETE=7,keystone:SELECT=91,keystone:INSERT=2 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:04:24.064898 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=89,keystone:INSERT=4,keystone:DELETE=7 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:04:24.093837 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:04:24.539622 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:04:24.665138 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=86,keystone:INSERT=4,keystone:DELETE=10 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:04:25.104803 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:04:25.482503 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:04:25.813275 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:04:25.968182 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=91,keystone:INSERT=3,keystone:DELETE=6 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:04:26.002592 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:04:26.789894 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:INSERT=2,keystone:DELETE=11,keystone:SELECT=87 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:04:26.910317 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:04:27.733803 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:04:27.941427 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:04:28.356879 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:04:28.590010 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:04:28.693772 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=93,keystone:INSERT=3,keystone:DELETE=4 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:04:30.009207 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:DELETE=16,keystone:SELECT=82,keystone:INSERT=2 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:04:30.693376 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c7ce8f3-5463-4526-84a1-d4b6b439fded None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:04:30.772621 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c7ce8f3-5463-4526-84a1-d4b6b439fded None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:04:30.777351 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93d8965e-9f19-4727-878e-e1ba59b86023 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:04:30.778071 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93d8965e-9f19-4727-878e-e1ba59b86023 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:04:30.787347 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93d8965e-9f19-4727-878e-e1ba59b86023 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:04:30.787347 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93d8965e-9f19-4727-878e-e1ba59b86023 admin admin] PATH_INFO: `/v3/users/c5e1193d61454052a68a493a8fe896a3` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:04:30.787347 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93d8965e-9f19-4727-878e-e1ba59b86023 admin admin] RBAC: Authorizing `identity:delete_user(user_id=c5e1193d61454052a68a493a8fe896a3)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:04:30.787347 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93d8965e-9f19-4727-878e-e1ba59b86023 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:04:30.943074 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-93d8965e-9f19-4727-878e-e1ba59b86023 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'c5e1193d61454052a68a493a8fe896a3'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:04:30.944185 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-93d8965e-9f19-4727-878e-e1ba59b86023 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'c5e1193d61454052a68a493a8fe896a3'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:04:30.944819 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-93d8965e-9f19-4727-878e-e1ba59b86023 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'c5e1193d61454052a68a493a8fe896a3'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:04:30.956906 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-93d8965e-9f19-4727-878e-e1ba59b86023 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'c5e1193d61454052a68a493a8fe896a3'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:04:30.964565 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-93d8965e-9f19-4727-878e-e1ba59b86023 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'c5e1193d61454052a68a493a8fe896a3'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:04:30.997178 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:04:31.064436 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1159/2315] 10.222.0.27 () {58 vars in 1224 bytes} [Tue Aug 30 02:04:30 2022] DELETE /identity/v3/users/c5e1193d61454052a68a493a8fe896a3 => generated 0 bytes in 372 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 02:04:31.069227 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dd897d2-d108-4ff3-a524-6b7638406437 None None] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:04:31.069726 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dd897d2-d108-4ff3-a524-6b7638406437 None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:04:31.069830 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dd897d2-d108-4ff3-a524-6b7638406437 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:04:31.089584 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.auth.core [None req-0dd897d2-d108-4ff3-a524-6b7638406437 None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81578) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:04:31.091056 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-0dd897d2-d108-4ff3-a524-6b7638406437 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:04:31.150747 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1157/2316] 10.222.0.27 () {58 vars in 958 bytes} [Tue Aug 30 02:04:31 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 30 02:04:31.160456 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93d8965e-9f19-4727-878e-e1ba59b86023 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:04:31.162462 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-93d8965e-9f19-4727-878e-e1ba59b86023 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:04:31.218567 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93d8965e-9f19-4727-878e-e1ba59b86023 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:04:31.222217 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e70f2e8e-fa20-42f9-91fa-09a7063cf4f0 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f3ea1580c56f4814a92bbb2acf542905', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4562f3951fbe465280073e60a11e44e2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:04:31.222812 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e70f2e8e-fa20-42f9-91fa-09a7063cf4f0 service neutron] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:04:31.223087 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e70f2e8e-fa20-42f9-91fa-09a7063cf4f0 service neutron] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:04:31.223352 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e70f2e8e-fa20-42f9-91fa-09a7063cf4f0 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:04:31.232967 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e70f2e8e-fa20-42f9-91fa-09a7063cf4f0 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:04:31.234478 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e70f2e8e-fa20-42f9-91fa-09a7063cf4f0 service neutron] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:04:31.244222 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1160/2317] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Aug 30 02:04:31 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 02:04:31.865166 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0dd897d2-d108-4ff3-a524-6b7638406437 None None] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:04:31.866621 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-0dd897d2-d108-4ff3-a524-6b7638406437 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:04:31.878915 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0dd897d2-d108-4ff3-a524-6b7638406437 None None] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:04:31.884175 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c961c40-14e6-4c21-b93e-524d3670eade admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:04:31.885001 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c961c40-14e6-4c21-b93e-524d3670eade admin admin] REQUEST_METHOD: `DELETE` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:04:31.885431 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c961c40-14e6-4c21-b93e-524d3670eade admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:04:31.885877 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c961c40-14e6-4c21-b93e-524d3670eade admin admin] PATH_INFO: `/v3/projects/a6412607e3e34ed98f036a3f34805345` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:04:31.893138 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c961c40-14e6-4c21-b93e-524d3670eade admin admin] RBAC: Authorizing `identity:delete_project(project_id=a6412607e3e34ed98f036a3f34805345)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:04:31.893138 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c961c40-14e6-4c21-b93e-524d3670eade admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:04:32.043647 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-9c961c40-14e6-4c21-b93e-524d3670eade admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'a6412607e3e34ed98f036a3f34805345', 'request_id': 'req-9c961c40-14e6-4c21-b93e-524d3670eade'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:04:32.050289 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-9c961c40-14e6-4c21-b93e-524d3670eade admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'a6412607e3e34ed98f036a3f34805345', 'request_id': 'req-9c961c40-14e6-4c21-b93e-524d3670eade'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:04:32.057328 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-9c961c40-14e6-4c21-b93e-524d3670eade admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'a6412607e3e34ed98f036a3f34805345', 'request_id': 'req-9c961c40-14e6-4c21-b93e-524d3670eade'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:04:32.261279 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-9c961c40-14e6-4c21-b93e-524d3670eade admin admin] The token cache is being invalidate because project a6412607e3e34ed98f036a3f34805345 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81578) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 30 02:04:32.261279 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-9c961c40-14e6-4c21-b93e-524d3670eade admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:04:32.261279 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-9c961c40-14e6-4c21-b93e-524d3670eade admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:04:32.261279 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1158/2318] 10.222.0.27 () {58 vars in 1233 bytes} [Tue Aug 30 02:04:31 2022] DELETE /identity/v3/projects/a6412607e3e34ed98f036a3f34805345 => generated 0 bytes in 393 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 02:04:32.502744 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e70f2e8e-fa20-42f9-91fa-09a7063cf4f0 service neutron] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:04:32.504241 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-e70f2e8e-fa20-42f9-91fa-09a7063cf4f0 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:04:32.517364 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e70f2e8e-fa20-42f9-91fa-09a7063cf4f0 service neutron] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:04:32.557199 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:04:32.575779 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a04ac5c7-0b2f-4a36-ad27-60bc35f725a7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:04:32.576404 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a04ac5c7-0b2f-4a36-ad27-60bc35f725a7 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:04:32.576752 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a04ac5c7-0b2f-4a36-ad27-60bc35f725a7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:04:32.576964 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a04ac5c7-0b2f-4a36-ad27-60bc35f725a7 admin admin] PATH_INFO: `/v3/users/cd95476d9b054391a02a08caa89238a4` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:04:32.578153 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a04ac5c7-0b2f-4a36-ad27-60bc35f725a7 admin admin] RBAC: Authorizing `identity:delete_user(user_id=cd95476d9b054391a02a08caa89238a4)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:04:32.580093 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a04ac5c7-0b2f-4a36-ad27-60bc35f725a7 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:04:32.795892 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-a04ac5c7-0b2f-4a36-ad27-60bc35f725a7 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'cd95476d9b054391a02a08caa89238a4'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:04:32.796714 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-a04ac5c7-0b2f-4a36-ad27-60bc35f725a7 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'cd95476d9b054391a02a08caa89238a4'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:04:32.797432 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-a04ac5c7-0b2f-4a36-ad27-60bc35f725a7 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'cd95476d9b054391a02a08caa89238a4'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:04:32.811837 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-a04ac5c7-0b2f-4a36-ad27-60bc35f725a7 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'cd95476d9b054391a02a08caa89238a4'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:04:32.819510 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-a04ac5c7-0b2f-4a36-ad27-60bc35f725a7 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'cd95476d9b054391a02a08caa89238a4'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:04:33.019142 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1161/2319] 10.222.0.27 () {58 vars in 1224 bytes} [Tue Aug 30 02:04:32 2022] DELETE /identity/v3/users/cd95476d9b054391a02a08caa89238a4 => generated 0 bytes in 517 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 02:04:33.024104 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e60dceaf-cee0-4ca8-998f-47ad3395b25a None None] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:04:33.024393 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e60dceaf-cee0-4ca8-998f-47ad3395b25a None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:04:33.024669 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e60dceaf-cee0-4ca8-998f-47ad3395b25a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:04:33.045083 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.auth.core [None req-e60dceaf-cee0-4ca8-998f-47ad3395b25a None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81578) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:04:33.046353 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-e60dceaf-cee0-4ca8-998f-47ad3395b25a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:04:33.115143 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1159/2320] 10.222.0.27 () {58 vars in 958 bytes} [Tue Aug 30 02:04:33 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 02:04:33.128371 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a04ac5c7-0b2f-4a36-ad27-60bc35f725a7 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:04:33.131019 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-a04ac5c7-0b2f-4a36-ad27-60bc35f725a7 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:04:33.182406 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e60dceaf-cee0-4ca8-998f-47ad3395b25a None None] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:04:33.184613 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-e60dceaf-cee0-4ca8-998f-47ad3395b25a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:04:33.210580 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e60dceaf-cee0-4ca8-998f-47ad3395b25a None None] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:04:33.211987 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a04ac5c7-0b2f-4a36-ad27-60bc35f725a7 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:04:33.217281 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cb199e9-60d3-43ce-baf5-c8db8a45d0d6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:04:33.217976 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cb199e9-60d3-43ce-baf5-c8db8a45d0d6 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:04:33.218085 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfcd3d76-36a0-469d-b825-e7cd4fe8d5e7 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f3ea1580c56f4814a92bbb2acf542905', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4562f3951fbe465280073e60a11e44e2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:04:33.218253 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cb199e9-60d3-43ce-baf5-c8db8a45d0d6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:04:33.218705 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cb199e9-60d3-43ce-baf5-c8db8a45d0d6 admin admin] PATH_INFO: `/v3/users/7b8810c558074c4cb91d414cae952fdc` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:04:33.218843 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfcd3d76-36a0-469d-b825-e7cd4fe8d5e7 service neutron] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:04:33.219109 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfcd3d76-36a0-469d-b825-e7cd4fe8d5e7 service neutron] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:04:33.219388 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfcd3d76-36a0-469d-b825-e7cd4fe8d5e7 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:04:33.220811 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2cb199e9-60d3-43ce-baf5-c8db8a45d0d6 admin admin] RBAC: Authorizing `identity:delete_user(user_id=7b8810c558074c4cb91d414cae952fdc)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:04:33.222904 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2cb199e9-60d3-43ce-baf5-c8db8a45d0d6 admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:04:33.234679 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfcd3d76-36a0-469d-b825-e7cd4fe8d5e7 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:04:33.236430 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfcd3d76-36a0-469d-b825-e7cd4fe8d5e7 service neutron] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:04:33.252015 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1162/2321] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Aug 30 02:04:33 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 124 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 02:04:33.268337 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=91,keystone:DELETE=8,keystone:INSERT=1 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:04:33.566968 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-2cb199e9-60d3-43ce-baf5-c8db8a45d0d6 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '7b8810c558074c4cb91d414cae952fdc'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:04:33.567691 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-2cb199e9-60d3-43ce-baf5-c8db8a45d0d6 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '7b8810c558074c4cb91d414cae952fdc'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:04:33.687933 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-2cb199e9-60d3-43ce-baf5-c8db8a45d0d6 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '7b8810c558074c4cb91d414cae952fdc'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:04:33.688670 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-2cb199e9-60d3-43ce-baf5-c8db8a45d0d6 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '7b8810c558074c4cb91d414cae952fdc'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:04:33.697705 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-2cb199e9-60d3-43ce-baf5-c8db8a45d0d6 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '7b8810c558074c4cb91d414cae952fdc'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:04:33.720325 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1160/2322] 10.222.0.27 () {58 vars in 1224 bytes} [Tue Aug 30 02:04:33 2022] DELETE /identity/v3/users/7b8810c558074c4cb91d414cae952fdc => generated 0 bytes in 539 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 30 02:04:33.727907 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3ec090d-1b60-4287-8172-f7406729a52e None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:04:33.728308 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3ec090d-1b60-4287-8172-f7406729a52e None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:04:33.728467 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3ec090d-1b60-4287-8172-f7406729a52e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:04:33.750196 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-e3ec090d-1b60-4287-8172-f7406729a52e None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:04:33.751401 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-e3ec090d-1b60-4287-8172-f7406729a52e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:04:33.816125 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1163/2323] 10.222.0.27 () {58 vars in 958 bytes} [Tue Aug 30 02:04:33 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 02:04:33.826894 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cb199e9-60d3-43ce-baf5-c8db8a45d0d6 admin admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:04:33.828350 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-2cb199e9-60d3-43ce-baf5-c8db8a45d0d6 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:04:33.890660 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cb199e9-60d3-43ce-baf5-c8db8a45d0d6 admin admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:04:33.892504 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3ec090d-1b60-4287-8172-f7406729a52e None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:04:33.895551 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e85e6877-eef3-4e26-a8ee-b85d27a61afb service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f3ea1580c56f4814a92bbb2acf542905', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4562f3951fbe465280073e60a11e44e2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:04:33.896011 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-e3ec090d-1b60-4287-8172-f7406729a52e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:04:33.896596 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e85e6877-eef3-4e26-a8ee-b85d27a61afb service neutron] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:04:33.896880 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e85e6877-eef3-4e26-a8ee-b85d27a61afb service neutron] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:04:33.897229 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e85e6877-eef3-4e26-a8ee-b85d27a61afb service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:04:33.905290 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:04:33.913353 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e85e6877-eef3-4e26-a8ee-b85d27a61afb service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:04:33.913598 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3ec090d-1b60-4287-8172-f7406729a52e None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:04:33.915734 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e85e6877-eef3-4e26-a8ee-b85d27a61afb service neutron] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:04:33.921606 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-794a17fb-6c7f-4ff2-8883-8934a07334b3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:04:33.922254 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-794a17fb-6c7f-4ff2-8883-8934a07334b3 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:04:33.922535 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-794a17fb-6c7f-4ff2-8883-8934a07334b3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:04:33.922985 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-794a17fb-6c7f-4ff2-8883-8934a07334b3 admin admin] PATH_INFO: `/v3/projects/7ebe8393a20f421ba3a7e6f26d5d2696` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:04:33.924134 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-794a17fb-6c7f-4ff2-8883-8934a07334b3 admin admin] RBAC: Authorizing `identity:delete_project(project_id=7ebe8393a20f421ba3a7e6f26d5d2696)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:04:33.926471 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-794a17fb-6c7f-4ff2-8883-8934a07334b3 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:04:33.927353 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1161/2324] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Aug 30 02:04:33 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 101 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 02:04:34.065998 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-794a17fb-6c7f-4ff2-8883-8934a07334b3 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '7ebe8393a20f421ba3a7e6f26d5d2696', 'request_id': 'req-794a17fb-6c7f-4ff2-8883-8934a07334b3'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:04:34.071651 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-794a17fb-6c7f-4ff2-8883-8934a07334b3 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '7ebe8393a20f421ba3a7e6f26d5d2696', 'request_id': 'req-794a17fb-6c7f-4ff2-8883-8934a07334b3'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:04:34.082465 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-794a17fb-6c7f-4ff2-8883-8934a07334b3 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '7ebe8393a20f421ba3a7e6f26d5d2696', 'request_id': 'req-794a17fb-6c7f-4ff2-8883-8934a07334b3'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:04:34.258598 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-794a17fb-6c7f-4ff2-8883-8934a07334b3 admin admin] The token cache is being invalidate because project 7ebe8393a20f421ba3a7e6f26d5d2696 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81577) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 30 02:04:34.258916 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-794a17fb-6c7f-4ff2-8883-8934a07334b3 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:04:34.259543 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-794a17fb-6c7f-4ff2-8883-8934a07334b3 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:04:34.261160 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1164/2325] 10.222.0.27 () {58 vars in 1233 bytes} [Tue Aug 30 02:04:33 2022] DELETE /identity/v3/projects/7ebe8393a20f421ba3a7e6f26d5d2696 => generated 0 bytes in 371 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 02:04:34.922684 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e85e6877-eef3-4e26-a8ee-b85d27a61afb service neutron] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:04:34.924172 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-e85e6877-eef3-4e26-a8ee-b85d27a61afb service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:04:34.946670 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e85e6877-eef3-4e26-a8ee-b85d27a61afb service neutron] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:04:35.040671 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20a7ca39-0eee-4830-9a1c-3627100daca1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:04:35.041677 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20a7ca39-0eee-4830-9a1c-3627100daca1 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:04:35.041980 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20a7ca39-0eee-4830-9a1c-3627100daca1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:04:35.042418 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20a7ca39-0eee-4830-9a1c-3627100daca1 admin admin] PATH_INFO: `/v3/projects/fb4e4792c04e48ab9f1fc222832612be` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:04:35.044046 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20a7ca39-0eee-4830-9a1c-3627100daca1 admin admin] RBAC: Authorizing `identity:delete_project(project_id=fb4e4792c04e48ab9f1fc222832612be)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:04:35.046143 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20a7ca39-0eee-4830-9a1c-3627100daca1 admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:04:35.219702 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:DELETE=12,keystone:SELECT=87,keystone:INSERT=1 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:04:35.232710 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-20a7ca39-0eee-4830-9a1c-3627100daca1 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'fb4e4792c04e48ab9f1fc222832612be', 'request_id': 'req-20a7ca39-0eee-4830-9a1c-3627100daca1'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:04:35.240679 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-20a7ca39-0eee-4830-9a1c-3627100daca1 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'fb4e4792c04e48ab9f1fc222832612be', 'request_id': 'req-20a7ca39-0eee-4830-9a1c-3627100daca1'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:04:35.249236 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-20a7ca39-0eee-4830-9a1c-3627100daca1 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'fb4e4792c04e48ab9f1fc222832612be', 'request_id': 'req-20a7ca39-0eee-4830-9a1c-3627100daca1'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:04:35.360088 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-20a7ca39-0eee-4830-9a1c-3627100daca1 admin admin] The token cache is being invalidate because project fb4e4792c04e48ab9f1fc222832612be was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81578) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 30 02:04:35.360372 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-20a7ca39-0eee-4830-9a1c-3627100daca1 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:04:35.360956 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-20a7ca39-0eee-4830-9a1c-3627100daca1 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:04:35.371535 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1162/2326] 10.222.0.27 () {58 vars in 1233 bytes} [Tue Aug 30 02:04:34 2022] DELETE /identity/v3/projects/fb4e4792c04e48ab9f1fc222832612be => generated 0 bytes in 449 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 02:04:35.376713 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc1d2104-abb5-4601-8687-f3f3ad191717 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:04:35.377895 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc1d2104-abb5-4601-8687-f3f3ad191717 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:04:35.378353 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc1d2104-abb5-4601-8687-f3f3ad191717 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:04:35.399947 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-bc1d2104-abb5-4601-8687-f3f3ad191717 None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:04:35.401145 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-bc1d2104-abb5-4601-8687-f3f3ad191717 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:04:35.477306 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7444a478-08f4-4cf2-9a3d-09d681c996a5 None None] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:04:35.477306 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7444a478-08f4-4cf2-9a3d-09d681c996a5 None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:04:35.477306 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7444a478-08f4-4cf2-9a3d-09d681c996a5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:04:35.482888 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1165/2327] 10.222.0.27 () {58 vars in 958 bytes} [Tue Aug 30 02:04:35 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 02:04:35.495638 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c687665d-562d-4501-a9b9-1a2f8910f3c4 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:04:35.496257 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c687665d-562d-4501-a9b9-1a2f8910f3c4 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:04:35.496770 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c687665d-562d-4501-a9b9-1a2f8910f3c4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:04:35.499481 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.auth.core [None req-7444a478-08f4-4cf2-9a3d-09d681c996a5 None None] MFA Rules not processed for user `5e7584fe83684b01baa31ab67edac026`. Rule list: `[]` (Enabled: `True`). {{(pid=81578) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:04:35.500743 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-7444a478-08f4-4cf2-9a3d-09d681c996a5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:04:35.524266 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-c687665d-562d-4501-a9b9-1a2f8910f3c4 None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:04:35.525841 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-c687665d-562d-4501-a9b9-1a2f8910f3c4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:04:35.535328 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1166/2328] 10.222.0.27 () {58 vars in 958 bytes} [Tue Aug 30 02:04:35 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 42 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 02:04:35.540279 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c687665d-562d-4501-a9b9-1a2f8910f3c4 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:04:35.551001 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c687665d-562d-4501-a9b9-1a2f8910f3c4 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:04:35.557035 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-413069c9-958e-4b79-aa6c-47640a0f5541 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:04:35.557686 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-413069c9-958e-4b79-aa6c-47640a0f5541 admin admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:04:35.557971 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-413069c9-958e-4b79-aa6c-47640a0f5541 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:04:35.558242 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-413069c9-958e-4b79-aa6c-47640a0f5541 admin admin] PATH_INFO: `/v3/domains` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:04:35.558901 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-413069c9-958e-4b79-aa6c-47640a0f5541 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 02:04:35.559209 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-413069c9-958e-4b79-aa6c-47640a0f5541 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:04:35.560815 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-413069c9-958e-4b79-aa6c-47640a0f5541 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:04:35.576268 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1167/2329] 10.222.0.27 () {58 vars in 1153 bytes} [Tue Aug 30 02:04:35 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 30 02:04:35.580289 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-413069c9-958e-4b79-aa6c-47640a0f5541 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:04:35.588567 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1163/2330] 10.222.0.27 () {58 vars in 958 bytes} [Tue Aug 30 02:04:35 2022] POST /identity/v3/auth/tokens => generated 3268 bytes in 113 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 30 02:04:35.594593 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-413069c9-958e-4b79-aa6c-47640a0f5541 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:04:35.600780 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0550d7fe-0976-4f49-9ccd-a4dca54e50c3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:04:35.601613 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0550d7fe-0976-4f49-9ccd-a4dca54e50c3 admin admin] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:04:35.602667 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0550d7fe-0976-4f49-9ccd-a4dca54e50c3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:04:35.603117 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0550d7fe-0976-4f49-9ccd-a4dca54e50c3 admin admin] PATH_INFO: `/v3/projects` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:04:35.603901 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0550d7fe-0976-4f49-9ccd-a4dca54e50c3 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:04:35.606071 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0550d7fe-0976-4f49-9ccd-a4dca54e50c3 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:04:35.728177 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1168/2331] 10.222.0.27 () {60 vars in 1153 bytes} [Tue Aug 30 02:04:35 2022] POST /identity/v3/projects => generated 381 bytes in 148 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 30 02:04:35.732370 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7444a478-08f4-4cf2-9a3d-09d681c996a5 None None] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:04:35.749503 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7444a478-08f4-4cf2-9a3d-09d681c996a5 None None] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:04:35.820334 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54e330ec-a8ba-428a-bc2f-2ae62db17d7c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:04:35.821013 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54e330ec-a8ba-428a-bc2f-2ae62db17d7c admin admin] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:04:35.821301 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54e330ec-a8ba-428a-bc2f-2ae62db17d7c admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:04:35.821637 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54e330ec-a8ba-428a-bc2f-2ae62db17d7c admin admin] PATH_INFO: `/v3/users` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:04:35.822286 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54e330ec-a8ba-428a-bc2f-2ae62db17d7c admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:04:35.824290 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54e330ec-a8ba-428a-bc2f-2ae62db17d7c admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:04:35.984518 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1164/2332] 10.222.0.27 () {60 vars in 1144 bytes} [Tue Aug 30 02:04:35 2022] POST /identity/v3/users => generated 291 bytes in 252 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 02:04:35.988481 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0550d7fe-0976-4f49-9ccd-a4dca54e50c3 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:04:35.999206 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0550d7fe-0976-4f49-9ccd-a4dca54e50c3 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:04:36.003986 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53a82986-beee-46dc-967d-f307635ddb7f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:04:36.004971 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53a82986-beee-46dc-967d-f307635ddb7f admin admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:04:36.005421 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53a82986-beee-46dc-967d-f307635ddb7f admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:04:36.005840 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53a82986-beee-46dc-967d-f307635ddb7f admin admin] PATH_INFO: `/v3/roles` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:04:36.006753 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53a82986-beee-46dc-967d-f307635ddb7f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:04:36.008833 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53a82986-beee-46dc-967d-f307635ddb7f admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:04:36.023174 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1169/2333] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Aug 30 02:04:35 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 30 02:04:36.036762 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54e330ec-a8ba-428a-bc2f-2ae62db17d7c admin admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:04:36.049156 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54e330ec-a8ba-428a-bc2f-2ae62db17d7c admin admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:04:36.054564 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b13b1b9-8338-442e-8c92-a080cec554a1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:04:36.055407 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b13b1b9-8338-442e-8c92-a080cec554a1 admin admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:04:36.055685 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b13b1b9-8338-442e-8c92-a080cec554a1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:04:36.056108 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b13b1b9-8338-442e-8c92-a080cec554a1 admin admin] PATH_INFO: `/v3/roles` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:04:36.056902 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b13b1b9-8338-442e-8c92-a080cec554a1 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:04:36.059310 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b13b1b9-8338-442e-8c92-a080cec554a1 admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:04:36.071308 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1165/2334] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Aug 30 02:04:36 2022] GET /identity/v3/roles => generated 1420 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 02:04:36.075828 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53a82986-beee-46dc-967d-f307635ddb7f admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:04:36.093761 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4a45c7c-2ab8-4335-aa13-e21bc443f9d2 None None] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:04:36.094265 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4a45c7c-2ab8-4335-aa13-e21bc443f9d2 None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:04:36.094844 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4a45c7c-2ab8-4335-aa13-e21bc443f9d2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:04:36.096469 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53a82986-beee-46dc-967d-f307635ddb7f admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:04:36.102855 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c87fd54-af40-45ed-beca-0e1971289b50 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:04:36.103747 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c87fd54-af40-45ed-beca-0e1971289b50 admin admin] REQUEST_METHOD: `PUT` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:04:36.104132 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c87fd54-af40-45ed-beca-0e1971289b50 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:04:36.104509 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c87fd54-af40-45ed-beca-0e1971289b50 admin admin] PATH_INFO: `/v3/projects/b6200be9acd14756908c3fd33adf1506/users/95c7f09cc2ba4dddae564bbe99ea8a7d/roles/1fb81afcb9f245f3b989add94598a063` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:04:36.125657 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c87fd54-af40-45ed-beca-0e1971289b50 admin admin] RBAC: Authorizing `identity:create_grant(project_id=b6200be9acd14756908c3fd33adf1506, user_id=95c7f09cc2ba4dddae564bbe99ea8a7d, role_id=1fb81afcb9f245f3b989add94598a063)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:04:36.128489 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c87fd54-af40-45ed-beca-0e1971289b50 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:04:36.138624 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.auth.core [None req-a4a45c7c-2ab8-4335-aa13-e21bc443f9d2 None None] MFA Rules not processed for user `7af1c1123b2e458b84248865287bd00e`. Rule list: `[]` (Enabled: `True`). {{(pid=81578) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:04:36.140394 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-a4a45c7c-2ab8-4335-aa13-e21bc443f9d2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:04:36.199583 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1166/2335] 10.222.0.27 () {58 vars in 958 bytes} [Tue Aug 30 02:04:36 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 30 02:04:36.294777 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1170/2336] 10.222.0.27 () {60 vars in 1483 bytes} [Tue Aug 30 02:04:36 2022] PUT /identity/v3/projects/b6200be9acd14756908c3fd33adf1506/users/95c7f09cc2ba4dddae564bbe99ea8a7d/roles/1fb81afcb9f245f3b989add94598a063 => generated 0 bytes in 219 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 02:04:36.300025 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed54f2ff-97a2-484b-b639-fe044804846e None None] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:04:36.300426 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed54f2ff-97a2-484b-b639-fe044804846e None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:04:36.300806 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed54f2ff-97a2-484b-b639-fe044804846e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:04:36.333704 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.auth.core [None req-ed54f2ff-97a2-484b-b639-fe044804846e None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81578) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:04:36.338860 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-ed54f2ff-97a2-484b-b639-fe044804846e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:04:36.416552 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1167/2337] 10.222.0.27 () {58 vars in 958 bytes} [Tue Aug 30 02:04:36 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 117 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 30 02:04:36.426605 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c87fd54-af40-45ed-beca-0e1971289b50 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:04:36.428966 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-6c87fd54-af40-45ed-beca-0e1971289b50 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:04:36.440363 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:04:36.494179 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c87fd54-af40-45ed-beca-0e1971289b50 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:04:36.499178 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29ce219e-6f98-4d55-b7ec-64d3c699545c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f3ea1580c56f4814a92bbb2acf542905', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4562f3951fbe465280073e60a11e44e2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:04:36.500210 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29ce219e-6f98-4d55-b7ec-64d3c699545c service neutron] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:04:36.500500 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29ce219e-6f98-4d55-b7ec-64d3c699545c service neutron] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:04:36.500776 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29ce219e-6f98-4d55-b7ec-64d3c699545c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:04:36.516552 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29ce219e-6f98-4d55-b7ec-64d3c699545c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:04:36.518333 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29ce219e-6f98-4d55-b7ec-64d3c699545c service neutron] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:04:36.532511 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1171/2338] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Aug 30 02:04:36 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 106 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 02:04:38.230394 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=92,keystone:DELETE=4,keystone:INSERT=4 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:04:44.577503 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10fea107-5cf3-45d9-8ffe-f1db814fa014 None None] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:04:44.578183 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10fea107-5cf3-45d9-8ffe-f1db814fa014 None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:04:44.578183 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10fea107-5cf3-45d9-8ffe-f1db814fa014 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:04:44.623474 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.auth.core [None req-10fea107-5cf3-45d9-8ffe-f1db814fa014 None None] MFA Rules not processed for user `95c7f09cc2ba4dddae564bbe99ea8a7d`. Rule list: `[]` (Enabled: `True`). {{(pid=81578) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:04:44.624823 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-10fea107-5cf3-45d9-8ffe-f1db814fa014 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:04:44.745342 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1168/2339] 10.222.0.27 () {58 vars in 958 bytes} [Tue Aug 30 02:04:44 2022] POST /identity/v3/auth/tokens => generated 3258 bytes in 169 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 02:04:44.746569 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1d26a25-c03f-47da-98f0-f5c65f862c79 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:04:44.747383 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1d26a25-c03f-47da-98f0-f5c65f862c79 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:04:44.748010 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1d26a25-c03f-47da-98f0-f5c65f862c79 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:04:44.751947 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10fea107-5cf3-45d9-8ffe-f1db814fa014 None None] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:04:44.779529 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-f1d26a25-c03f-47da-98f0-f5c65f862c79 None None] MFA Rules not processed for user `121c374d49a64335ab74a431cfbc1320`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:04:44.780954 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-f1d26a25-c03f-47da-98f0-f5c65f862c79 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:04:44.785735 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10fea107-5cf3-45d9-8ffe-f1db814fa014 None None] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:04:44.802947 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a535ae18-03f9-4549-91d5-c66465ef38c6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:04:44.809383 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a535ae18-03f9-4549-91d5-c66465ef38c6 admin admin] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:04:44.809383 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a535ae18-03f9-4549-91d5-c66465ef38c6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:04:44.809383 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a535ae18-03f9-4549-91d5-c66465ef38c6 admin admin] PATH_INFO: `/v3/projects` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:04:44.809383 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a535ae18-03f9-4549-91d5-c66465ef38c6 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:04:44.809383 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a535ae18-03f9-4549-91d5-c66465ef38c6 admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:04:44.893737 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1172/2340] 10.222.0.27 () {58 vars in 1009 bytes} [Tue Aug 30 02:04:44 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 149 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 30 02:04:44.953909 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1d26a25-c03f-47da-98f0-f5c65f862c79 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:04:44.955580 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-f1d26a25-c03f-47da-98f0-f5c65f862c79 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:04:45.004507 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1169/2341] 10.222.0.27 () {60 vars in 1153 bytes} [Tue Aug 30 02:04:44 2022] POST /identity/v3/projects => generated 383 bytes in 253 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 02:04:45.006181 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a535ae18-03f9-4549-91d5-c66465ef38c6 admin admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:04:45.066372 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1d26a25-c03f-47da-98f0-f5c65f862c79 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:04:45.089862 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a535ae18-03f9-4549-91d5-c66465ef38c6 admin admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:04:45.095962 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be186641-35bd-46e7-82b7-5afa1ef5756a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:04:45.096373 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be186641-35bd-46e7-82b7-5afa1ef5756a admin admin] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:04:45.096737 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be186641-35bd-46e7-82b7-5afa1ef5756a admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:04:45.097085 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be186641-35bd-46e7-82b7-5afa1ef5756a admin admin] PATH_INFO: `/v3/users` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:04:45.097905 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be186641-35bd-46e7-82b7-5afa1ef5756a admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:04:45.100277 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be186641-35bd-46e7-82b7-5afa1ef5756a admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:04:45.112294 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:04:45.151212 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:04:45.162514 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b006bfe-c058-4c29-8c01-bf1a4aeeebc3 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b4f0008fde424f22bc1f7307cf422774', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4562f3951fbe465280073e60a11e44e2', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:04:45.163137 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b006bfe-c058-4c29-8c01-bf1a4aeeebc3 service swift] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:04:45.163420 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b006bfe-c058-4c29-8c01-bf1a4aeeebc3 service swift] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:04:45.163688 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b006bfe-c058-4c29-8c01-bf1a4aeeebc3 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:04:45.247931 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b006bfe-c058-4c29-8c01-bf1a4aeeebc3 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:04:45.249672 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b006bfe-c058-4c29-8c01-bf1a4aeeebc3 service swift] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:04:45.259838 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1173/2342] 10.222.0.27 () {60 vars in 1384 bytes} [Tue Aug 30 02:04:44 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 307 msecs (HTTP/1.1 200) 6 headers in 379 bytes (2 switches on core 0) Aug 30 02:04:45.294680 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1170/2343] 10.222.0.27 () {60 vars in 1144 bytes} [Tue Aug 30 02:04:45 2022] POST /identity/v3/users => generated 298 bytes in 289 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 02:04:45.298400 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b006bfe-c058-4c29-8c01-bf1a4aeeebc3 service swift] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:04:45.313512 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b006bfe-c058-4c29-8c01-bf1a4aeeebc3 service swift] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:04:45.322250 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74562f40-f134-45c0-9633-6185f5398789 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:04:45.323006 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74562f40-f134-45c0-9633-6185f5398789 admin admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:04:45.323639 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74562f40-f134-45c0-9633-6185f5398789 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:04:45.323989 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74562f40-f134-45c0-9633-6185f5398789 admin admin] PATH_INFO: `/v3/roles` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:04:45.324756 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74562f40-f134-45c0-9633-6185f5398789 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:04:45.326503 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74562f40-f134-45c0-9633-6185f5398789 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:04:45.338449 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1174/2344] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Aug 30 02:04:45 2022] GET /identity/v3/roles => generated 1420 bytes in 41 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 30 02:04:45.343015 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be186641-35bd-46e7-82b7-5afa1ef5756a admin admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:04:45.360191 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be186641-35bd-46e7-82b7-5afa1ef5756a admin admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:04:45.366160 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b5fa78d-7f90-4834-8b41-95555bdd6838 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:04:45.366878 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b5fa78d-7f90-4834-8b41-95555bdd6838 admin admin] REQUEST_METHOD: `PUT` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:04:45.367165 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b5fa78d-7f90-4834-8b41-95555bdd6838 admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:04:45.367445 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b5fa78d-7f90-4834-8b41-95555bdd6838 admin admin] PATH_INFO: `/v3/projects/28f99120ebae4b29820cc0430ae94f59/users/b2927586e099498a91c07560ea4b234c/roles/ead5fac86a0747ea8823a82efcea8ed7` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:04:45.387855 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b5fa78d-7f90-4834-8b41-95555bdd6838 admin admin] RBAC: Authorizing `identity:create_grant(project_id=28f99120ebae4b29820cc0430ae94f59, user_id=b2927586e099498a91c07560ea4b234c, role_id=ead5fac86a0747ea8823a82efcea8ed7)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:04:45.390690 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b5fa78d-7f90-4834-8b41-95555bdd6838 admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:04:45.596588 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1171/2345] 10.222.0.27 () {60 vars in 1483 bytes} [Tue Aug 30 02:04:45 2022] PUT /identity/v3/projects/28f99120ebae4b29820cc0430ae94f59/users/b2927586e099498a91c07560ea4b234c/roles/ead5fac86a0747ea8823a82efcea8ed7 => generated 0 bytes in 251 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 30 02:04:45.603965 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74562f40-f134-45c0-9633-6185f5398789 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:04:45.614315 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74562f40-f134-45c0-9633-6185f5398789 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:04:45.688152 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28183775-511c-4d43-81bf-96a514729ad6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:04:45.689133 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28183775-511c-4d43-81bf-96a514729ad6 admin admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:04:45.689626 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28183775-511c-4d43-81bf-96a514729ad6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:04:45.690062 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28183775-511c-4d43-81bf-96a514729ad6 admin admin] PATH_INFO: `/v3/roles` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:04:45.690725 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28183775-511c-4d43-81bf-96a514729ad6 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:04:45.692439 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28183775-511c-4d43-81bf-96a514729ad6 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:04:45.703920 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1175/2346] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Aug 30 02:04:45 2022] GET /identity/v3/roles => generated 1420 bytes in 107 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 02:04:45.708560 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b5fa78d-7f90-4834-8b41-95555bdd6838 admin admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:04:45.727338 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b5fa78d-7f90-4834-8b41-95555bdd6838 admin admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:04:45.733178 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e660d74-7ba5-40b6-a6e6-1be543bc98d3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:04:45.734779 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e660d74-7ba5-40b6-a6e6-1be543bc98d3 admin admin] REQUEST_METHOD: `PUT` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:04:45.734779 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e660d74-7ba5-40b6-a6e6-1be543bc98d3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:04:45.734779 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e660d74-7ba5-40b6-a6e6-1be543bc98d3 admin admin] PATH_INFO: `/v3/projects/28f99120ebae4b29820cc0430ae94f59/users/b2927586e099498a91c07560ea4b234c/roles/ead5fac86a0747ea8823a82efcea8ed7` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:04:45.736829 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e660d74-7ba5-40b6-a6e6-1be543bc98d3 admin admin] RBAC: Authorizing `identity:create_grant(project_id=28f99120ebae4b29820cc0430ae94f59, user_id=b2927586e099498a91c07560ea4b234c, role_id=ead5fac86a0747ea8823a82efcea8ed7)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:04:45.740024 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e660d74-7ba5-40b6-a6e6-1be543bc98d3 admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:04:45.781921 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1172/2347] 10.222.0.27 () {60 vars in 1483 bytes} [Tue Aug 30 02:04:45 2022] PUT /identity/v3/projects/28f99120ebae4b29820cc0430ae94f59/users/b2927586e099498a91c07560ea4b234c/roles/ead5fac86a0747ea8823a82efcea8ed7 => generated 0 bytes in 70 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 02:04:46.729716 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-420fb65b-393b-4ccb-b6c8-ac0d5a3aae89 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:04:46.730242 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-420fb65b-393b-4ccb-b6c8-ac0d5a3aae89 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:04:46.730242 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-420fb65b-393b-4ccb-b6c8-ac0d5a3aae89 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:04:46.758613 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-420fb65b-393b-4ccb-b6c8-ac0d5a3aae89 None None] MFA Rules not processed for user `121c374d49a64335ab74a431cfbc1320`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:04:46.763796 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-420fb65b-393b-4ccb-b6c8-ac0d5a3aae89 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:04:46.870097 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1176/2348] 10.222.0.27 () {58 vars in 967 bytes} [Tue Aug 30 02:04:46 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 140 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 02:04:46.879110 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e660d74-7ba5-40b6-a6e6-1be543bc98d3 admin admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:04:46.991699 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e660d74-7ba5-40b6-a6e6-1be543bc98d3 admin admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:04:46.998444 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa42ba97-4efd-4f6d-b988-c5cbb662e062 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b4f0008fde424f22bc1f7307cf422774', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4562f3951fbe465280073e60a11e44e2', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:04:46.999284 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa42ba97-4efd-4f6d-b988-c5cbb662e062 service swift] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:04:46.999896 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa42ba97-4efd-4f6d-b988-c5cbb662e062 service swift] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:04:47.000367 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa42ba97-4efd-4f6d-b988-c5cbb662e062 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:04:47.011108 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa42ba97-4efd-4f6d-b988-c5cbb662e062 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:04:47.014377 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa42ba97-4efd-4f6d-b988-c5cbb662e062 service swift] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:04:47.026072 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1173/2349] 10.222.0.27 () {60 vars in 1384 bytes} [Tue Aug 30 02:04:46 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 149 msecs (HTTP/1.1 200) 6 headers in 379 bytes (2 switches on core 0) Aug 30 02:04:48.075924 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f6f5d9a-c6fa-4483-b6b8-e7b2c3cd0b7a None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:04:48.076363 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f6f5d9a-c6fa-4483-b6b8-e7b2c3cd0b7a None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:04:48.076619 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f6f5d9a-c6fa-4483-b6b8-e7b2c3cd0b7a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:04:48.104133 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-2f6f5d9a-c6fa-4483-b6b8-e7b2c3cd0b7a None None] MFA Rules not processed for user `121c374d49a64335ab74a431cfbc1320`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:04:48.105834 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-2f6f5d9a-c6fa-4483-b6b8-e7b2c3cd0b7a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:04:48.116349 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1177/2350] 10.222.0.27 () {58 vars in 1009 bytes} [Tue Aug 30 02:04:48 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 41 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 02:04:48.130274 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa42ba97-4efd-4f6d-b988-c5cbb662e062 service swift] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:04:48.142058 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa42ba97-4efd-4f6d-b988-c5cbb662e062 service swift] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:04:48.148836 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d415106-0133-4069-8a10-2eb05843dac9 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b4f0008fde424f22bc1f7307cf422774', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4562f3951fbe465280073e60a11e44e2', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:04:48.149767 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d415106-0133-4069-8a10-2eb05843dac9 service swift] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:04:48.150203 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d415106-0133-4069-8a10-2eb05843dac9 service swift] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:04:48.150798 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d415106-0133-4069-8a10-2eb05843dac9 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:04:48.162440 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d415106-0133-4069-8a10-2eb05843dac9 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:04:48.164509 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d415106-0133-4069-8a10-2eb05843dac9 service swift] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:04:48.175350 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1174/2351] 10.222.0.27 () {60 vars in 1384 bytes} [Tue Aug 30 02:04:48 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 46 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Aug 30 02:04:49.155785 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ab0a175-eb37-4d4b-85c7-1998dc0d7acd None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:04:49.156257 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ab0a175-eb37-4d4b-85c7-1998dc0d7acd None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:04:49.156522 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ab0a175-eb37-4d4b-85c7-1998dc0d7acd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:04:49.178790 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-4ab0a175-eb37-4d4b-85c7-1998dc0d7acd None None] MFA Rules not processed for user `121c374d49a64335ab74a431cfbc1320`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:04:49.180015 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-4ab0a175-eb37-4d4b-85c7-1998dc0d7acd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:04:49.188115 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1178/2352] 10.222.0.27 () {58 vars in 967 bytes} [Tue Aug 30 02:04:49 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 02:04:49.202881 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d415106-0133-4069-8a10-2eb05843dac9 service swift] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:04:49.213565 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d415106-0133-4069-8a10-2eb05843dac9 service swift] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:04:49.219814 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17ee8f7f-62bc-447f-b96c-c16756737340 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b4f0008fde424f22bc1f7307cf422774', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4562f3951fbe465280073e60a11e44e2', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:04:49.220516 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17ee8f7f-62bc-447f-b96c-c16756737340 service swift] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:04:49.220852 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17ee8f7f-62bc-447f-b96c-c16756737340 service swift] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:04:49.221189 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17ee8f7f-62bc-447f-b96c-c16756737340 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:04:49.231442 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17ee8f7f-62bc-447f-b96c-c16756737340 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:04:49.233108 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17ee8f7f-62bc-447f-b96c-c16756737340 service swift] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:04:49.244614 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1175/2353] 10.222.0.27 () {60 vars in 1384 bytes} [Tue Aug 30 02:04:49 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 42 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Aug 30 02:04:53.314622 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c64eca5-e770-41d0-b481-96a43cb57a09 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:04:53.315544 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c64eca5-e770-41d0-b481-96a43cb57a09 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:04:53.315544 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c64eca5-e770-41d0-b481-96a43cb57a09 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:04:53.359241 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-4c64eca5-e770-41d0-b481-96a43cb57a09 None None] MFA Rules not processed for user `b2927586e099498a91c07560ea4b234c`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:04:53.360742 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-4c64eca5-e770-41d0-b481-96a43cb57a09 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:04:53.370339 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:04:53.465658 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1179/2354] 10.222.0.27 () {58 vars in 958 bytes} [Tue Aug 30 02:04:53 2022] POST /identity/v3/auth/tokens => generated 3327 bytes in 156 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 02:04:53.477151 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17ee8f7f-62bc-447f-b96c-c16756737340 service swift] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:04:53.478925 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-17ee8f7f-62bc-447f-b96c-c16756737340 service swift] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:04:53.556363 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17ee8f7f-62bc-447f-b96c-c16756737340 service swift] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:04:53.556363 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c55b55c0-3179-4da0-bcbb-8d0f2634e4b3 service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3caff62093d748f48596574836e12e42', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4562f3951fbe465280073e60a11e44e2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:04:53.556946 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c55b55c0-3179-4da0-bcbb-8d0f2634e4b3 service glance] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:04:53.557233 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c55b55c0-3179-4da0-bcbb-8d0f2634e4b3 service glance] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:04:53.557543 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c55b55c0-3179-4da0-bcbb-8d0f2634e4b3 service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:04:53.563534 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c64eca5-e770-41d0-b481-96a43cb57a09 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:04:53.566771 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-4c64eca5-e770-41d0-b481-96a43cb57a09 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:04:53.607519 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:INSERT=3,keystone:SELECT=97 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:04:53.649153 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c55b55c0-3179-4da0-bcbb-8d0f2634e4b3 service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:04:53.651365 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c55b55c0-3179-4da0-bcbb-8d0f2634e4b3 service glance] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:04:53.664667 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1176/2355] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Aug 30 02:04:53 2022] GET /identity/v3/auth/tokens => generated 3258 bytes in 188 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 02:04:53.689905 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c64eca5-e770-41d0-b481-96a43cb57a09 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:04:53.695516 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe5b6572-c537-4b99-aeec-7bf26291750b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:04:53.696162 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe5b6572-c537-4b99-aeec-7bf26291750b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:04:53.696440 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe5b6572-c537-4b99-aeec-7bf26291750b admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:04:53.696716 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe5b6572-c537-4b99-aeec-7bf26291750b admin admin] PATH_INFO: `/v3/users/dbce595e3bec45deb120a3d6fe6d6091` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:04:53.698198 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe5b6572-c537-4b99-aeec-7bf26291750b admin admin] RBAC: Authorizing `identity:delete_user(user_id=dbce595e3bec45deb120a3d6fe6d6091)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:04:53.701503 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe5b6572-c537-4b99-aeec-7bf26291750b admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:04:54.123821 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-205ae5ae-7e9e-4636-8ce4-70e5b52fa289 None None] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:04:54.124111 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-205ae5ae-7e9e-4636-8ce4-70e5b52fa289 None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:04:54.124386 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-205ae5ae-7e9e-4636-8ce4-70e5b52fa289 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:04:54.135748 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-fe5b6572-c537-4b99-aeec-7bf26291750b admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'dbce595e3bec45deb120a3d6fe6d6091'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:04:54.136424 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-fe5b6572-c537-4b99-aeec-7bf26291750b admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'dbce595e3bec45deb120a3d6fe6d6091'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:04:54.137034 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-fe5b6572-c537-4b99-aeec-7bf26291750b admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'dbce595e3bec45deb120a3d6fe6d6091'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:04:54.148603 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.auth.core [None req-205ae5ae-7e9e-4636-8ce4-70e5b52fa289 None None] MFA Rules not processed for user `121c374d49a64335ab74a431cfbc1320`. Rule list: `[]` (Enabled: `True`). {{(pid=81578) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:04:54.150013 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-fe5b6572-c537-4b99-aeec-7bf26291750b admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'dbce595e3bec45deb120a3d6fe6d6091'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:04:54.150273 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-205ae5ae-7e9e-4636-8ce4-70e5b52fa289 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:04:54.159404 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-fe5b6572-c537-4b99-aeec-7bf26291750b admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'dbce595e3bec45deb120a3d6fe6d6091'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:04:54.159978 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1177/2356] 10.222.0.27 () {58 vars in 1009 bytes} [Tue Aug 30 02:04:54 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 36 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 30 02:04:54.173372 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-205ae5ae-7e9e-4636-8ce4-70e5b52fa289 None None] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:04:54.175133 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-205ae5ae-7e9e-4636-8ce4-70e5b52fa289 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:04:54.190443 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-205ae5ae-7e9e-4636-8ce4-70e5b52fa289 None None] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:04:54.196118 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c24e8697-e7c8-48be-bdeb-dae0b62edb10 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b4f0008fde424f22bc1f7307cf422774', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4562f3951fbe465280073e60a11e44e2', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:04:54.196715 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c24e8697-e7c8-48be-bdeb-dae0b62edb10 service swift] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:04:54.197004 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c24e8697-e7c8-48be-bdeb-dae0b62edb10 service swift] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:04:54.197254 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c24e8697-e7c8-48be-bdeb-dae0b62edb10 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:04:54.205349 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c24e8697-e7c8-48be-bdeb-dae0b62edb10 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:04:54.206973 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c24e8697-e7c8-48be-bdeb-dae0b62edb10 service swift] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:04:54.215268 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1178/2357] 10.222.0.27 () {60 vars in 1384 bytes} [Tue Aug 30 02:04:54 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 42 msecs (HTTP/1.1 200) 6 headers in 379 bytes (2 switches on core 0) Aug 30 02:04:54.293656 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1180/2358] 10.222.0.27 () {58 vars in 1224 bytes} [Tue Aug 30 02:04:53 2022] DELETE /identity/v3/users/dbce595e3bec45deb120a3d6fe6d6091 => generated 0 bytes in 731 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 30 02:04:54.298011 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdb389b3-9ff5-4c26-9bb3-6b9a1183e846 None None] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:04:54.298909 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdb389b3-9ff5-4c26-9bb3-6b9a1183e846 None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:04:54.299187 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdb389b3-9ff5-4c26-9bb3-6b9a1183e846 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:04:54.318919 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.auth.core [None req-cdb389b3-9ff5-4c26-9bb3-6b9a1183e846 None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81578) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:04:54.321020 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-cdb389b3-9ff5-4c26-9bb3-6b9a1183e846 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:04:54.416324 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1179/2359] 10.222.0.27 () {58 vars in 958 bytes} [Tue Aug 30 02:04:54 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 119 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 02:04:54.427464 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe5b6572-c537-4b99-aeec-7bf26291750b admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:04:54.430039 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-fe5b6572-c537-4b99-aeec-7bf26291750b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:04:54.515480 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe5b6572-c537-4b99-aeec-7bf26291750b admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:04:54.559527 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:04:54.578760 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8e8d03c-5d9a-4b4e-855c-a189f99aa8ed service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f3ea1580c56f4814a92bbb2acf542905', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4562f3951fbe465280073e60a11e44e2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:04:54.579386 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8e8d03c-5d9a-4b4e-855c-a189f99aa8ed service neutron] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:04:54.579663 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8e8d03c-5d9a-4b4e-855c-a189f99aa8ed service neutron] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:04:54.579936 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8e8d03c-5d9a-4b4e-855c-a189f99aa8ed service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:04:54.632145 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8e8d03c-5d9a-4b4e-855c-a189f99aa8ed service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:04:54.634571 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8e8d03c-5d9a-4b4e-855c-a189f99aa8ed service neutron] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:04:54.648031 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1181/2360] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Aug 30 02:04:54 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 221 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 02:04:55.157567 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdb389b3-9ff5-4c26-9bb3-6b9a1183e846 None None] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:04:55.159485 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-cdb389b3-9ff5-4c26-9bb3-6b9a1183e846 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:04:55.181788 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdb389b3-9ff5-4c26-9bb3-6b9a1183e846 None None] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:04:55.195912 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7abe52f7-e035-4a8c-b123-bd712cc6328f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:04:55.196562 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7abe52f7-e035-4a8c-b123-bd712cc6328f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:04:55.197200 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7abe52f7-e035-4a8c-b123-bd712cc6328f admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:04:55.197669 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7abe52f7-e035-4a8c-b123-bd712cc6328f admin admin] PATH_INFO: `/v3/projects/d779875224e742f9a3f3bf4bfcf12fbc` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:04:55.202852 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7abe52f7-e035-4a8c-b123-bd712cc6328f admin admin] RBAC: Authorizing `identity:delete_project(project_id=d779875224e742f9a3f3bf4bfcf12fbc)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:04:55.213832 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7abe52f7-e035-4a8c-b123-bd712cc6328f admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:04:55.487166 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-7abe52f7-e035-4a8c-b123-bd712cc6328f admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'd779875224e742f9a3f3bf4bfcf12fbc', 'request_id': 'req-7abe52f7-e035-4a8c-b123-bd712cc6328f'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:04:55.509155 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-7abe52f7-e035-4a8c-b123-bd712cc6328f admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'd779875224e742f9a3f3bf4bfcf12fbc', 'request_id': 'req-7abe52f7-e035-4a8c-b123-bd712cc6328f'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:04:55.517440 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-7abe52f7-e035-4a8c-b123-bd712cc6328f admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'd779875224e742f9a3f3bf4bfcf12fbc', 'request_id': 'req-7abe52f7-e035-4a8c-b123-bd712cc6328f'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:04:55.667021 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-7abe52f7-e035-4a8c-b123-bd712cc6328f admin admin] The token cache is being invalidate because project d779875224e742f9a3f3bf4bfcf12fbc was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81578) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 30 02:04:55.667131 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-7abe52f7-e035-4a8c-b123-bd712cc6328f admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:04:55.670999 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-7abe52f7-e035-4a8c-b123-bd712cc6328f admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:04:55.673139 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1180/2361] 10.222.0.27 () {58 vars in 1233 bytes} [Tue Aug 30 02:04:55 2022] DELETE /identity/v3/projects/d779875224e742f9a3f3bf4bfcf12fbc => generated 0 bytes in 516 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 02:05:00.539601 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8e8d03c-5d9a-4b4e-855c-a189f99aa8ed service neutron] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:05:00.540930 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-d8e8d03c-5d9a-4b4e-855c-a189f99aa8ed service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:05:00.554935 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8e8d03c-5d9a-4b4e-855c-a189f99aa8ed service neutron] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:05:00.626362 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35d9a27a-e466-4a26-8687-3dd94a636f86 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:05:00.627011 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35d9a27a-e466-4a26-8687-3dd94a636f86 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:05:00.627306 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35d9a27a-e466-4a26-8687-3dd94a636f86 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:05:00.627566 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35d9a27a-e466-4a26-8687-3dd94a636f86 admin admin] PATH_INFO: `/v3/users/4e76bbd5d9234b938d4655dbe6fd6771` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:05:00.628867 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35d9a27a-e466-4a26-8687-3dd94a636f86 admin admin] RBAC: Authorizing `identity:delete_user(user_id=4e76bbd5d9234b938d4655dbe6fd6771)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:05:00.634518 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35d9a27a-e466-4a26-8687-3dd94a636f86 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:05:00.928912 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-35d9a27a-e466-4a26-8687-3dd94a636f86 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '4e76bbd5d9234b938d4655dbe6fd6771'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:05:00.929913 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-35d9a27a-e466-4a26-8687-3dd94a636f86 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '4e76bbd5d9234b938d4655dbe6fd6771'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:05:00.930695 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-35d9a27a-e466-4a26-8687-3dd94a636f86 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '4e76bbd5d9234b938d4655dbe6fd6771'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:05:00.952318 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-35d9a27a-e466-4a26-8687-3dd94a636f86 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '4e76bbd5d9234b938d4655dbe6fd6771'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:05:00.961833 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-35d9a27a-e466-4a26-8687-3dd94a636f86 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '4e76bbd5d9234b938d4655dbe6fd6771'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:05:01.075620 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1182/2362] 10.222.0.27 () {58 vars in 1224 bytes} [Tue Aug 30 02:05:00 2022] DELETE /identity/v3/users/4e76bbd5d9234b938d4655dbe6fd6771 => generated 0 bytes in 534 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 02:05:01.077904 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57434953-2f6d-49a8-84cd-1a4fa3b271dc None None] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:05:01.078420 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57434953-2f6d-49a8-84cd-1a4fa3b271dc None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:05:01.078537 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57434953-2f6d-49a8-84cd-1a4fa3b271dc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:05:01.113857 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.auth.core [None req-57434953-2f6d-49a8-84cd-1a4fa3b271dc None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81578) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:05:01.117008 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-57434953-2f6d-49a8-84cd-1a4fa3b271dc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:05:01.164241 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:05:01.210627 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1181/2363] 10.222.0.27 () {58 vars in 958 bytes} [Tue Aug 30 02:05:01 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 134 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 02:05:01.220398 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35d9a27a-e466-4a26-8687-3dd94a636f86 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:05:01.222008 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-35d9a27a-e466-4a26-8687-3dd94a636f86 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:05:01.294239 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:05:01.310599 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35d9a27a-e466-4a26-8687-3dd94a636f86 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:05:01.315004 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-697ae6a6-482c-4758-b8f9-3912e02a4da2 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f3ea1580c56f4814a92bbb2acf542905', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4562f3951fbe465280073e60a11e44e2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:05:01.315629 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-697ae6a6-482c-4758-b8f9-3912e02a4da2 service neutron] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:05:01.315907 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-697ae6a6-482c-4758-b8f9-3912e02a4da2 service neutron] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:05:01.316177 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-697ae6a6-482c-4758-b8f9-3912e02a4da2 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:05:01.329807 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-697ae6a6-482c-4758-b8f9-3912e02a4da2 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:05:01.331371 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-697ae6a6-482c-4758-b8f9-3912e02a4da2 service neutron] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:05:01.344990 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1183/2364] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Aug 30 02:05:01 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 125 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 02:05:02.088493 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57434953-2f6d-49a8-84cd-1a4fa3b271dc None None] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:05:02.090651 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-57434953-2f6d-49a8-84cd-1a4fa3b271dc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:05:02.134826 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57434953-2f6d-49a8-84cd-1a4fa3b271dc None None] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:05:02.161002 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17396e6e-8430-4077-a532-e09eb9b32e08 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:05:02.169891 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17396e6e-8430-4077-a532-e09eb9b32e08 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:05:02.171037 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17396e6e-8430-4077-a532-e09eb9b32e08 admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:05:02.172521 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17396e6e-8430-4077-a532-e09eb9b32e08 admin admin] PATH_INFO: `/v3/projects/3cd3d5ba273045e5a9fb3b29228887d6` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:05:02.177862 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17396e6e-8430-4077-a532-e09eb9b32e08 admin admin] RBAC: Authorizing `identity:delete_project(project_id=3cd3d5ba273045e5a9fb3b29228887d6)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:05:02.180098 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17396e6e-8430-4077-a532-e09eb9b32e08 admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:05:02.362002 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-17396e6e-8430-4077-a532-e09eb9b32e08 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '3cd3d5ba273045e5a9fb3b29228887d6', 'request_id': 'req-17396e6e-8430-4077-a532-e09eb9b32e08'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:05:02.373752 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-17396e6e-8430-4077-a532-e09eb9b32e08 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '3cd3d5ba273045e5a9fb3b29228887d6', 'request_id': 'req-17396e6e-8430-4077-a532-e09eb9b32e08'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:05:02.383552 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-17396e6e-8430-4077-a532-e09eb9b32e08 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '3cd3d5ba273045e5a9fb3b29228887d6', 'request_id': 'req-17396e6e-8430-4077-a532-e09eb9b32e08'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:05:02.551019 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-17396e6e-8430-4077-a532-e09eb9b32e08 admin admin] The token cache is being invalidate because project 3cd3d5ba273045e5a9fb3b29228887d6 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81578) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 30 02:05:02.556773 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-17396e6e-8430-4077-a532-e09eb9b32e08 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:05:02.556773 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-17396e6e-8430-4077-a532-e09eb9b32e08 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:05:02.556773 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1182/2365] 10.222.0.27 () {58 vars in 1233 bytes} [Tue Aug 30 02:05:02 2022] DELETE /identity/v3/projects/3cd3d5ba273045e5a9fb3b29228887d6 => generated 0 bytes in 467 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 02:05:02.564417 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32f67aa9-b26f-451e-bbf6-2e55b286d9d5 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:05:02.565039 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32f67aa9-b26f-451e-bbf6-2e55b286d9d5 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:05:02.565039 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32f67aa9-b26f-451e-bbf6-2e55b286d9d5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:05:02.615270 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-32f67aa9-b26f-451e-bbf6-2e55b286d9d5 None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:05:02.616623 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-32f67aa9-b26f-451e-bbf6-2e55b286d9d5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:05:02.727826 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1184/2366] 10.222.0.27 () {58 vars in 958 bytes} [Tue Aug 30 02:05:02 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 164 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 02:05:02.736191 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e06d327-6a31-4679-a911-fa619ffa3915 None None] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:05:02.737015 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e06d327-6a31-4679-a911-fa619ffa3915 None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:05:02.737361 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e06d327-6a31-4679-a911-fa619ffa3915 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:05:02.768521 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.auth.core [None req-3e06d327-6a31-4679-a911-fa619ffa3915 None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81578) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:05:02.770166 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-3e06d327-6a31-4679-a911-fa619ffa3915 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:05:02.779821 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1183/2367] 10.222.0.27 () {58 vars in 958 bytes} [Tue Aug 30 02:05:02 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 45 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 02:05:02.785075 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32f67aa9-b26f-451e-bbf6-2e55b286d9d5 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:05:02.798520 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32f67aa9-b26f-451e-bbf6-2e55b286d9d5 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:05:02.804380 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58844103-8075-4a1c-bfee-735cb8fa58de admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:05:02.804622 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58844103-8075-4a1c-bfee-735cb8fa58de admin admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:05:02.804904 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58844103-8075-4a1c-bfee-735cb8fa58de admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:05:02.805177 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58844103-8075-4a1c-bfee-735cb8fa58de admin admin] PATH_INFO: `/v3/domains` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:05:02.805868 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58844103-8075-4a1c-bfee-735cb8fa58de admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 02:05:02.806200 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58844103-8075-4a1c-bfee-735cb8fa58de admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:05:02.807818 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58844103-8075-4a1c-bfee-735cb8fa58de admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:05:02.820747 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1185/2368] 10.222.0.27 () {58 vars in 1153 bytes} [Tue Aug 30 02:05:02 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 02:05:02.830151 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e06d327-6a31-4679-a911-fa619ffa3915 None None] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:05:02.842639 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e06d327-6a31-4679-a911-fa619ffa3915 None None] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:05:02.848217 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e3db74f-4270-41c7-a5d2-25720c3fe5fa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:05:02.848919 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e3db74f-4270-41c7-a5d2-25720c3fe5fa admin admin] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:05:02.849259 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e3db74f-4270-41c7-a5d2-25720c3fe5fa admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:05:02.849612 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e3db74f-4270-41c7-a5d2-25720c3fe5fa admin admin] PATH_INFO: `/v3/projects` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:05:02.850324 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e3db74f-4270-41c7-a5d2-25720c3fe5fa admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:05:02.852272 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e3db74f-4270-41c7-a5d2-25720c3fe5fa admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:05:03.418971 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1184/2369] 10.222.0.27 () {60 vars in 1153 bytes} [Tue Aug 30 02:05:02 2022] POST /identity/v3/projects => generated 383 bytes in 594 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 02:05:03.422207 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58844103-8075-4a1c-bfee-735cb8fa58de admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:05:03.434424 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58844103-8075-4a1c-bfee-735cb8fa58de admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:05:03.547447 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-461a0344-e0c0-43c2-9404-1f91a9c4e82d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:05:03.548370 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-461a0344-e0c0-43c2-9404-1f91a9c4e82d admin admin] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:05:03.548707 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-461a0344-e0c0-43c2-9404-1f91a9c4e82d admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:05:03.549077 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-461a0344-e0c0-43c2-9404-1f91a9c4e82d admin admin] PATH_INFO: `/v3/users` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:05:03.549871 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-461a0344-e0c0-43c2-9404-1f91a9c4e82d admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:05:03.552112 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-461a0344-e0c0-43c2-9404-1f91a9c4e82d admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:05:03.707596 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e3db74f-4270-41c7-a5d2-25720c3fe5fa admin admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:05:03.714587 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1186/2370] 10.222.0.27 () {60 vars in 1144 bytes} [Tue Aug 30 02:05:03 2022] POST /identity/v3/users => generated 292 bytes in 295 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 30 02:05:03.721287 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e3db74f-4270-41c7-a5d2-25720c3fe5fa admin admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:05:03.728179 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f3cc492-c7e4-4192-8acf-ff1b8dafad8d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:05:03.730103 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f3cc492-c7e4-4192-8acf-ff1b8dafad8d admin admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:05:03.730550 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f3cc492-c7e4-4192-8acf-ff1b8dafad8d admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:05:03.730991 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f3cc492-c7e4-4192-8acf-ff1b8dafad8d admin admin] PATH_INFO: `/v3/roles` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:05:03.731980 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f3cc492-c7e4-4192-8acf-ff1b8dafad8d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:05:03.733786 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f3cc492-c7e4-4192-8acf-ff1b8dafad8d admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:05:03.750499 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1185/2371] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Aug 30 02:05:03 2022] GET /identity/v3/roles => generated 1420 bytes in 43 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 02:05:03.762378 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-461a0344-e0c0-43c2-9404-1f91a9c4e82d admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:05:03.776229 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-461a0344-e0c0-43c2-9404-1f91a9c4e82d admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:05:03.784422 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a024f8f1-bcc1-465b-8e4a-d4fcf8be436a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:05:03.784422 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a024f8f1-bcc1-465b-8e4a-d4fcf8be436a admin admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:05:03.784422 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a024f8f1-bcc1-465b-8e4a-d4fcf8be436a admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:05:03.784422 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a024f8f1-bcc1-465b-8e4a-d4fcf8be436a admin admin] PATH_INFO: `/v3/roles` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:05:03.785048 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a024f8f1-bcc1-465b-8e4a-d4fcf8be436a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:05:03.787107 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a024f8f1-bcc1-465b-8e4a-d4fcf8be436a admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:05:03.806722 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1187/2372] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Aug 30 02:05:03 2022] GET /identity/v3/roles => generated 1420 bytes in 44 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 02:05:03.809759 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f3cc492-c7e4-4192-8acf-ff1b8dafad8d admin admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:05:03.821279 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f3cc492-c7e4-4192-8acf-ff1b8dafad8d admin admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:05:03.827799 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf8078a8-b820-4cb6-9d08-5c446911d7a7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:05:03.828637 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf8078a8-b820-4cb6-9d08-5c446911d7a7 admin admin] REQUEST_METHOD: `PUT` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:05:03.828768 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf8078a8-b820-4cb6-9d08-5c446911d7a7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:05:03.828982 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf8078a8-b820-4cb6-9d08-5c446911d7a7 admin admin] PATH_INFO: `/v3/projects/072e2c259a164ecaa45b5ff18fa3ac66/users/789377907cfd4f0f9d3123f6418163f7/roles/1fb81afcb9f245f3b989add94598a063` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:05:03.847484 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf8078a8-b820-4cb6-9d08-5c446911d7a7 admin admin] RBAC: Authorizing `identity:create_grant(project_id=072e2c259a164ecaa45b5ff18fa3ac66, user_id=789377907cfd4f0f9d3123f6418163f7, role_id=1fb81afcb9f245f3b989add94598a063)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:05:03.850522 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf8078a8-b820-4cb6-9d08-5c446911d7a7 admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:05:03.963500 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1186/2373] 10.222.0.27 () {60 vars in 1483 bytes} [Tue Aug 30 02:05:03 2022] PUT /identity/v3/projects/072e2c259a164ecaa45b5ff18fa3ac66/users/789377907cfd4f0f9d3123f6418163f7/roles/1fb81afcb9f245f3b989add94598a063 => generated 0 bytes in 152 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 02:05:03.967831 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87e5055e-1be5-442e-bf60-2aa8a578cbc4 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:05:03.968230 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87e5055e-1be5-442e-bf60-2aa8a578cbc4 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:05:03.968612 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87e5055e-1be5-442e-bf60-2aa8a578cbc4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:05:04.013082 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-87e5055e-1be5-442e-bf60-2aa8a578cbc4 None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:05:04.014403 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-87e5055e-1be5-442e-bf60-2aa8a578cbc4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:05:04.154775 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1188/2374] 10.222.0.27 () {58 vars in 958 bytes} [Tue Aug 30 02:05:03 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 184 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 02:05:04.160232 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf8078a8-b820-4cb6-9d08-5c446911d7a7 admin admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:05:04.163257 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-cf8078a8-b820-4cb6-9d08-5c446911d7a7 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:05:04.290589 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf8078a8-b820-4cb6-9d08-5c446911d7a7 admin admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:05:04.295337 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76c5473f-9d09-486e-9101-3b218959e4dc service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f3ea1580c56f4814a92bbb2acf542905', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4562f3951fbe465280073e60a11e44e2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:05:04.296262 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76c5473f-9d09-486e-9101-3b218959e4dc service neutron] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:05:04.296848 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76c5473f-9d09-486e-9101-3b218959e4dc service neutron] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:05:04.297529 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76c5473f-9d09-486e-9101-3b218959e4dc service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:05:04.310777 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76c5473f-9d09-486e-9101-3b218959e4dc service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:05:04.312634 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76c5473f-9d09-486e-9101-3b218959e4dc service neutron] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:05:04.319169 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:05:04.329827 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1187/2375] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Aug 30 02:05:04 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 170 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 02:05:04.381520 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:05:06.468422 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74c7d4d5-99a9-476d-bc8e-cf4ccae15573 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:05:06.469380 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74c7d4d5-99a9-476d-bc8e-cf4ccae15573 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:05:06.487346 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74c7d4d5-99a9-476d-bc8e-cf4ccae15573 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:05:06.534675 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-74c7d4d5-99a9-476d-bc8e-cf4ccae15573 None None] MFA Rules not processed for user `789377907cfd4f0f9d3123f6418163f7`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:05:06.542287 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-74c7d4d5-99a9-476d-bc8e-cf4ccae15573 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:05:06.651477 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1189/2376] 10.222.0.27 () {58 vars in 958 bytes} [Tue Aug 30 02:05:06 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 184 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 02:05:06.663080 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76c5473f-9d09-486e-9101-3b218959e4dc service neutron] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:05:06.664676 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-76c5473f-9d09-486e-9101-3b218959e4dc service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:05:06.742261 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76c5473f-9d09-486e-9101-3b218959e4dc service neutron] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:05:06.748577 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6729467-4b6b-44c0-b1f5-d587834a12bc service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ab936b3767a8417aa80c8c8f1b84b667', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4562f3951fbe465280073e60a11e44e2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:05:06.749228 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6729467-4b6b-44c0-b1f5-d587834a12bc service cinder] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:05:06.749536 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6729467-4b6b-44c0-b1f5-d587834a12bc service cinder] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:05:06.749841 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6729467-4b6b-44c0-b1f5-d587834a12bc service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:05:06.759870 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6729467-4b6b-44c0-b1f5-d587834a12bc service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:05:06.761299 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6729467-4b6b-44c0-b1f5-d587834a12bc service cinder] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:05:06.770151 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1188/2377] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Aug 30 02:05:06 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 108 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 02:05:09.024928 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8819cc08-1034-4c88-8fb6-5990bb0ef8fc None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:05:09.025730 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8819cc08-1034-4c88-8fb6-5990bb0ef8fc None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:05:09.025730 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8819cc08-1034-4c88-8fb6-5990bb0ef8fc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:05:09.045905 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-8819cc08-1034-4c88-8fb6-5990bb0ef8fc None None] MFA Rules not processed for user `95c7f09cc2ba4dddae564bbe99ea8a7d`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:05:09.047212 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-8819cc08-1034-4c88-8fb6-5990bb0ef8fc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:05:09.105968 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1190/2378] 10.222.0.27 () {58 vars in 958 bytes} [Tue Aug 30 02:05:09 2022] POST /identity/v3/auth/tokens => generated 3258 bytes in 82 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 02:05:09.640747 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-baea3dd0-280e-41a2-a7d8-4d9d6cb7dc75 None None] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:05:09.641364 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-baea3dd0-280e-41a2-a7d8-4d9d6cb7dc75 None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:05:09.641475 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-baea3dd0-280e-41a2-a7d8-4d9d6cb7dc75 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:05:09.663268 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.auth.core [None req-baea3dd0-280e-41a2-a7d8-4d9d6cb7dc75 None None] MFA Rules not processed for user `789377907cfd4f0f9d3123f6418163f7`. Rule list: `[]` (Enabled: `True`). {{(pid=81578) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:05:09.663732 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-baea3dd0-280e-41a2-a7d8-4d9d6cb7dc75 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:05:09.671224 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1189/2379] 10.222.0.27 () {58 vars in 958 bytes} [Tue Aug 30 02:05:09 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 02:05:11.973593 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8819cc08-1034-4c88-8fb6-5990bb0ef8fc None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:05:11.974970 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-8819cc08-1034-4c88-8fb6-5990bb0ef8fc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:05:12.045734 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8819cc08-1034-4c88-8fb6-5990bb0ef8fc None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:05:12.051440 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1ec7bf6-45cf-4536-92e6-3f19a9900d94 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '379536ff06fd4f719899e5dde732e63d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4562f3951fbe465280073e60a11e44e2', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:05:12.051998 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1ec7bf6-45cf-4536-92e6-3f19a9900d94 service placement] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:05:12.052275 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1ec7bf6-45cf-4536-92e6-3f19a9900d94 service placement] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:05:12.052623 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1ec7bf6-45cf-4536-92e6-3f19a9900d94 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:05:12.054909 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-e1ec7bf6-45cf-4536-92e6-3f19a9900d94 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:05:12.067947 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1ec7bf6-45cf-4536-92e6-3f19a9900d94 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:05:12.070419 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1ec7bf6-45cf-4536-92e6-3f19a9900d94 service placement] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:05:12.081366 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1191/2380] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Aug 30 02:05:11 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 108 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 02:05:14.545756 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-721b74d8-b564-4312-8ee5-47137b8a730f None None] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:05:14.546237 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-721b74d8-b564-4312-8ee5-47137b8a730f None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:05:14.549541 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-721b74d8-b564-4312-8ee5-47137b8a730f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:05:14.568087 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.auth.core [None req-721b74d8-b564-4312-8ee5-47137b8a730f None None] MFA Rules not processed for user `121c374d49a64335ab74a431cfbc1320`. Rule list: `[]` (Enabled: `True`). {{(pid=81578) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:05:14.569229 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-721b74d8-b564-4312-8ee5-47137b8a730f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:05:14.629019 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1190/2381] 10.222.0.27 () {58 vars in 1009 bytes} [Tue Aug 30 02:05:14 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 30 02:05:14.642103 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1ec7bf6-45cf-4536-92e6-3f19a9900d94 service placement] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:05:14.644780 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-e1ec7bf6-45cf-4536-92e6-3f19a9900d94 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:05:14.695656 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:05:14.752429 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1ec7bf6-45cf-4536-92e6-3f19a9900d94 service placement] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:05:14.758167 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd8189de-ba68-4006-8dbc-8f60abc4189b service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b4f0008fde424f22bc1f7307cf422774', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4562f3951fbe465280073e60a11e44e2', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:05:14.758782 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd8189de-ba68-4006-8dbc-8f60abc4189b service swift] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:05:14.759063 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd8189de-ba68-4006-8dbc-8f60abc4189b service swift] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:05:14.759333 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd8189de-ba68-4006-8dbc-8f60abc4189b service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:05:14.768228 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd8189de-ba68-4006-8dbc-8f60abc4189b service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:05:14.769857 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd8189de-ba68-4006-8dbc-8f60abc4189b service swift] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:05:14.782824 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1192/2382] 10.222.0.27 () {60 vars in 1384 bytes} [Tue Aug 30 02:05:14 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 138 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Aug 30 02:05:24.621437 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=54 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:05:25.505865 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=24 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:05:58.665270 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-721b74d8-b564-4312-8ee5-47137b8a730f None None] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:05:58.666748 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-721b74d8-b564-4312-8ee5-47137b8a730f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:05:58.678455 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-721b74d8-b564-4312-8ee5-47137b8a730f None None] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:05:58.682013 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecc9267f-f355-442d-a86f-a2e0b2d19174 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f3ea1580c56f4814a92bbb2acf542905', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4562f3951fbe465280073e60a11e44e2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:05:58.682620 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecc9267f-f355-442d-a86f-a2e0b2d19174 service neutron] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:05:58.682898 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecc9267f-f355-442d-a86f-a2e0b2d19174 service neutron] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:05:58.683201 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecc9267f-f355-442d-a86f-a2e0b2d19174 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:05:58.684938 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-ecc9267f-f355-442d-a86f-a2e0b2d19174 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:05:58.751748 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ecc9267f-f355-442d-a86f-a2e0b2d19174 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:05:58.753163 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ecc9267f-f355-442d-a86f-a2e0b2d19174 service neutron] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:05:58.763829 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1191/2383] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Aug 30 02:05:58 2022] GET /identity/v3/auth/tokens => generated 3327 bytes in 99 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 02:06:08.758984 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=30 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:06:09.486873 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ca3ef7d-362e-443f-bc80-e6ce4cebe4b3 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:06:09.487378 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ca3ef7d-362e-443f-bc80-e6ce4cebe4b3 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:06:09.487475 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ca3ef7d-362e-443f-bc80-e6ce4cebe4b3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:06:09.507116 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-9ca3ef7d-362e-443f-bc80-e6ce4cebe4b3 None None] MFA Rules not processed for user `7af1c1123b2e458b84248865287bd00e`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:06:09.508320 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-9ca3ef7d-362e-443f-bc80-e6ce4cebe4b3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:06:09.562847 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1193/2384] 10.222.0.27 () {58 vars in 958 bytes} [Tue Aug 30 02:06:09 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 77 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 02:06:17.665630 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef769c1e-3055-458d-90e6-f3e0886c65f1 None None] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:06:17.666157 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef769c1e-3055-458d-90e6-f3e0886c65f1 None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:06:17.666157 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef769c1e-3055-458d-90e6-f3e0886c65f1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:06:17.686012 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.auth.core [None req-ef769c1e-3055-458d-90e6-f3e0886c65f1 None None] MFA Rules not processed for user `5e7584fe83684b01baa31ab67edac026`. Rule list: `[]` (Enabled: `True`). {{(pid=81578) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:06:17.687212 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-ef769c1e-3055-458d-90e6-f3e0886c65f1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:06:17.742115 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1192/2385] 10.222.0.27 () {58 vars in 958 bytes} [Tue Aug 30 02:06:17 2022] POST /identity/v3/auth/tokens => generated 3268 bytes in 78 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 02:06:19.555603 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=25 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:06:27.735433 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=25 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:06:43.913728 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75d21641-332d-449c-96c3-89a3eddd1df9 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:06:43.914261 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75d21641-332d-449c-96c3-89a3eddd1df9 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:06:43.914386 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75d21641-332d-449c-96c3-89a3eddd1df9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:06:43.937488 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-75d21641-332d-449c-96c3-89a3eddd1df9 None None] MFA Rules not processed for user `789377907cfd4f0f9d3123f6418163f7`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:06:43.938900 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-75d21641-332d-449c-96c3-89a3eddd1df9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:06:43.946308 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1194/2386] 10.222.0.27 () {58 vars in 958 bytes} [Tue Aug 30 02:06:43 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 02:06:49.495099 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23a24e82-0cd8-45f2-9d75-463c1e8b8108 None None] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:06:49.495099 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23a24e82-0cd8-45f2-9d75-463c1e8b8108 None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:06:49.495571 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23a24e82-0cd8-45f2-9d75-463c1e8b8108 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:06:49.515826 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.auth.core [None req-23a24e82-0cd8-45f2-9d75-463c1e8b8108 None None] MFA Rules not processed for user `bda980735a1f4948ba695e6c2788f3b4`. Rule list: `[]` (Enabled: `True`). {{(pid=81578) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:06:49.517027 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-23a24e82-0cd8-45f2-9d75-463c1e8b8108 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:06:49.579029 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1193/2387] 10.222.0.27 () {58 vars in 958 bytes} [Tue Aug 30 02:06:49 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 02:06:53.930408 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=3 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:06:58.084588 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75d21641-332d-449c-96c3-89a3eddd1df9 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:06:58.094862 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75d21641-332d-449c-96c3-89a3eddd1df9 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:06:58.098681 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6e37c46-df87-4e8b-8f95-3f44b2e29678 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f3ea1580c56f4814a92bbb2acf542905', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4562f3951fbe465280073e60a11e44e2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:06:58.099422 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6e37c46-df87-4e8b-8f95-3f44b2e29678 service neutron] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:06:58.099699 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6e37c46-df87-4e8b-8f95-3f44b2e29678 service neutron] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:06:58.099968 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6e37c46-df87-4e8b-8f95-3f44b2e29678 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:06:58.101705 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-a6e37c46-df87-4e8b-8f95-3f44b2e29678 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:06:58.172223 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6e37c46-df87-4e8b-8f95-3f44b2e29678 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:06:58.173676 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6e37c46-df87-4e8b-8f95-3f44b2e29678 service neutron] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:06:58.184352 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1195/2388] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Aug 30 02:06:58 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 100 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 02:06:58.706695 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23a24e82-0cd8-45f2-9d75-463c1e8b8108 None None] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:06:58.717843 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23a24e82-0cd8-45f2-9d75-463c1e8b8108 None None] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:06:58.723342 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1acdca6c-4005-46ed-a01f-d6ee4ca63372 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f3ea1580c56f4814a92bbb2acf542905', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4562f3951fbe465280073e60a11e44e2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:06:58.723964 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1acdca6c-4005-46ed-a01f-d6ee4ca63372 service neutron] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:06:58.724239 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1acdca6c-4005-46ed-a01f-d6ee4ca63372 service neutron] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:06:58.724508 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1acdca6c-4005-46ed-a01f-d6ee4ca63372 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:06:58.726373 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-1acdca6c-4005-46ed-a01f-d6ee4ca63372 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:06:58.742122 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1acdca6c-4005-46ed-a01f-d6ee4ca63372 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:06:58.743581 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1acdca6c-4005-46ed-a01f-d6ee4ca63372 service neutron] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:06:58.758210 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1194/2389] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Aug 30 02:06:58 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 52 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 30 02:07:08.179567 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=34 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:07:08.751961 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=31 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:07:12.290578 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6e37c46-df87-4e8b-8f95-3f44b2e29678 service neutron] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:07:12.291899 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-a6e37c46-df87-4e8b-8f95-3f44b2e29678 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:07:12.305803 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6e37c46-df87-4e8b-8f95-3f44b2e29678 service neutron] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:07:12.311421 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07bee65e-dc53-4768-99c2-7f2bc2652895 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd64f2e71361644589bec8a06e6eceed1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4562f3951fbe465280073e60a11e44e2', 'project_domain_id': 'default', 'roles': ['member', 'service', 'admin', 'ResellerAdmin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:07:12.312108 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07bee65e-dc53-4768-99c2-7f2bc2652895 service nova] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:07:12.312345 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07bee65e-dc53-4768-99c2-7f2bc2652895 service nova] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:07:12.312628 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07bee65e-dc53-4768-99c2-7f2bc2652895 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:07:12.314303 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-07bee65e-dc53-4768-99c2-7f2bc2652895 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:07:12.324153 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07bee65e-dc53-4768-99c2-7f2bc2652895 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:07:12.326270 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07bee65e-dc53-4768-99c2-7f2bc2652895 service nova] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:07:12.335537 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1196/2390] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Aug 30 02:07:12 2022] GET /identity/v3/auth/tokens => generated 3268 bytes in 46 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 02:07:15.056016 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1acdca6c-4005-46ed-a01f-d6ee4ca63372 service neutron] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:07:15.065365 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1acdca6c-4005-46ed-a01f-d6ee4ca63372 service neutron] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:07:15.069061 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efbf87ea-be01-4607-baf4-832bde78c570 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ab936b3767a8417aa80c8c8f1b84b667', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4562f3951fbe465280073e60a11e44e2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:07:15.069683 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efbf87ea-be01-4607-baf4-832bde78c570 service cinder] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:07:15.069962 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efbf87ea-be01-4607-baf4-832bde78c570 service cinder] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:07:15.070233 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efbf87ea-be01-4607-baf4-832bde78c570 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:07:15.071929 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-efbf87ea-be01-4607-baf4-832bde78c570 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:07:15.084169 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efbf87ea-be01-4607-baf4-832bde78c570 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:07:15.088048 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efbf87ea-be01-4607-baf4-832bde78c570 service cinder] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:07:15.095726 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1195/2391] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Aug 30 02:07:15 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 40 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 02:07:22.331164 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=6 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:07:25.090654 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=6 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:07:37.345228 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07bee65e-dc53-4768-99c2-7f2bc2652895 service nova] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:07:37.346719 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-07bee65e-dc53-4768-99c2-7f2bc2652895 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:07:37.361335 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07bee65e-dc53-4768-99c2-7f2bc2652895 service nova] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:07:37.366926 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84bd8658-3a9c-4a4b-918e-77a551be3022 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd64f2e71361644589bec8a06e6eceed1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4562f3951fbe465280073e60a11e44e2', 'project_domain_id': 'default', 'roles': ['member', 'service', 'admin', 'ResellerAdmin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:07:37.367502 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84bd8658-3a9c-4a4b-918e-77a551be3022 service nova] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:07:37.367797 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84bd8658-3a9c-4a4b-918e-77a551be3022 service nova] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:07:37.368048 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84bd8658-3a9c-4a4b-918e-77a551be3022 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:07:37.369741 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-84bd8658-3a9c-4a4b-918e-77a551be3022 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:07:37.383079 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84bd8658-3a9c-4a4b-918e-77a551be3022 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:07:37.384622 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84bd8658-3a9c-4a4b-918e-77a551be3022 service nova] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:07:37.394995 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1197/2392] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Aug 30 02:07:37 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 50 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 02:07:40.926025 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efbf87ea-be01-4607-baf4-832bde78c570 service cinder] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:07:40.933586 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efbf87ea-be01-4607-baf4-832bde78c570 service cinder] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:07:40.937944 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cf8c559-5155-45b8-9394-993831df2347 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f3ea1580c56f4814a92bbb2acf542905', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4562f3951fbe465280073e60a11e44e2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:07:40.938727 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cf8c559-5155-45b8-9394-993831df2347 service neutron] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:07:40.939009 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cf8c559-5155-45b8-9394-993831df2347 service neutron] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:07:40.939304 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cf8c559-5155-45b8-9394-993831df2347 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:07:40.941264 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-4cf8c559-5155-45b8-9394-993831df2347 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:07:40.955282 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4cf8c559-5155-45b8-9394-993831df2347 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:07:40.956964 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4cf8c559-5155-45b8-9394-993831df2347 service neutron] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:07:40.967659 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1196/2393] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Aug 30 02:07:40 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 45 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 02:07:45.043527 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84bd8658-3a9c-4a4b-918e-77a551be3022 service nova] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:07:45.045585 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-84bd8658-3a9c-4a4b-918e-77a551be3022 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:07:45.061527 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84bd8658-3a9c-4a4b-918e-77a551be3022 service nova] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:07:45.067561 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7416eb80-fb56-4a5e-aafe-66056c4bb2da admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:07:45.071598 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7416eb80-fb56-4a5e-aafe-66056c4bb2da admin admin] REQUEST_METHOD: `DELETE` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:07:45.071598 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7416eb80-fb56-4a5e-aafe-66056c4bb2da admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:07:45.071598 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7416eb80-fb56-4a5e-aafe-66056c4bb2da admin admin] PATH_INFO: `/v3/users/5e7584fe83684b01baa31ab67edac026` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:07:45.071598 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7416eb80-fb56-4a5e-aafe-66056c4bb2da admin admin] RBAC: Authorizing `identity:delete_user(user_id=5e7584fe83684b01baa31ab67edac026)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:07:45.072596 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7416eb80-fb56-4a5e-aafe-66056c4bb2da admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:07:45.313752 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-7416eb80-fb56-4a5e-aafe-66056c4bb2da admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '5e7584fe83684b01baa31ab67edac026'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:07:45.314410 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-7416eb80-fb56-4a5e-aafe-66056c4bb2da admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '5e7584fe83684b01baa31ab67edac026'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:07:45.315041 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-7416eb80-fb56-4a5e-aafe-66056c4bb2da admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '5e7584fe83684b01baa31ab67edac026'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:07:45.327777 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-7416eb80-fb56-4a5e-aafe-66056c4bb2da admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '5e7584fe83684b01baa31ab67edac026'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:07:45.334783 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-7416eb80-fb56-4a5e-aafe-66056c4bb2da admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '5e7584fe83684b01baa31ab67edac026'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:07:45.377361 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb2121eb-7d10-40a7-845c-6a63fe0097f1 None None] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:07:45.377942 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb2121eb-7d10-40a7-845c-6a63fe0097f1 None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:07:45.377942 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb2121eb-7d10-40a7-845c-6a63fe0097f1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:07:45.398171 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.auth.core [None req-bb2121eb-7d10-40a7-845c-6a63fe0097f1 None None] MFA Rules not processed for user `7af1c1123b2e458b84248865287bd00e`. Rule list: `[]` (Enabled: `True`). {{(pid=81578) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:07:45.409119 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-bb2121eb-7d10-40a7-845c-6a63fe0097f1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:07:45.409119 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1197/2394] 10.222.0.27 () {58 vars in 958 bytes} [Tue Aug 30 02:07:45 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 30 02:07:45.437866 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1198/2395] 10.222.0.27 () {58 vars in 1224 bytes} [Tue Aug 30 02:07:45 2022] DELETE /identity/v3/users/5e7584fe83684b01baa31ab67edac026 => generated 0 bytes in 395 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 02:07:45.820796 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb2121eb-7d10-40a7-845c-6a63fe0097f1 None None] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:07:45.822648 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-bb2121eb-7d10-40a7-845c-6a63fe0097f1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:07:45.835309 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb2121eb-7d10-40a7-845c-6a63fe0097f1 None None] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:07:45.875328 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7416eb80-fb56-4a5e-aafe-66056c4bb2da admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:07:45.877162 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-7416eb80-fb56-4a5e-aafe-66056c4bb2da admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:07:45.903033 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79d87b92-d60d-4a76-af8a-4f88e3d95bde service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f3ea1580c56f4814a92bbb2acf542905', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4562f3951fbe465280073e60a11e44e2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:07:45.905010 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79d87b92-d60d-4a76-af8a-4f88e3d95bde service neutron] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:07:45.905010 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79d87b92-d60d-4a76-af8a-4f88e3d95bde service neutron] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:07:45.905010 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79d87b92-d60d-4a76-af8a-4f88e3d95bde service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:07:45.908297 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-79d87b92-d60d-4a76-af8a-4f88e3d95bde service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:07:45.922706 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79d87b92-d60d-4a76-af8a-4f88e3d95bde service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:07:45.924173 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79d87b92-d60d-4a76-af8a-4f88e3d95bde service neutron] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:07:45.934168 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1198/2396] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Aug 30 02:07:45 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 114 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 02:07:45.970086 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7416eb80-fb56-4a5e-aafe-66056c4bb2da admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:07:45.974929 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0384174-340c-4493-82ca-90c37aa8a6e6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:07:45.975721 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0384174-340c-4493-82ca-90c37aa8a6e6 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:07:45.976567 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0384174-340c-4493-82ca-90c37aa8a6e6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:07:45.977044 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0384174-340c-4493-82ca-90c37aa8a6e6 admin admin] PATH_INFO: `/v3/projects/e7f4b9bbe0b640688754d04484900cca` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:07:45.978745 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0384174-340c-4493-82ca-90c37aa8a6e6 admin admin] RBAC: Authorizing `identity:delete_project(project_id=e7f4b9bbe0b640688754d04484900cca)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:07:45.981858 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0384174-340c-4493-82ca-90c37aa8a6e6 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:07:46.056575 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79d87b92-d60d-4a76-af8a-4f88e3d95bde service neutron] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:07:46.058299 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-79d87b92-d60d-4a76-af8a-4f88e3d95bde service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:07:46.138780 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-c0384174-340c-4493-82ca-90c37aa8a6e6 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'e7f4b9bbe0b640688754d04484900cca', 'request_id': 'req-c0384174-340c-4493-82ca-90c37aa8a6e6'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:07:46.148953 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79d87b92-d60d-4a76-af8a-4f88e3d95bde service neutron] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:07:46.152083 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-c0384174-340c-4493-82ca-90c37aa8a6e6 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'e7f4b9bbe0b640688754d04484900cca', 'request_id': 'req-c0384174-340c-4493-82ca-90c37aa8a6e6'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:07:46.163388 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=83,keystone:DELETE=16,keystone:INSERT=1 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:07:46.168938 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-c0384174-340c-4493-82ca-90c37aa8a6e6 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'e7f4b9bbe0b640688754d04484900cca', 'request_id': 'req-c0384174-340c-4493-82ca-90c37aa8a6e6'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:07:46.269078 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6f34a90-78ed-437f-96b0-cabd2005c0a1 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd64f2e71361644589bec8a06e6eceed1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4562f3951fbe465280073e60a11e44e2', 'project_domain_id': 'default', 'roles': ['member', 'service', 'admin', 'ResellerAdmin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:07:46.269736 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6f34a90-78ed-437f-96b0-cabd2005c0a1 service nova] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:07:46.270036 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6f34a90-78ed-437f-96b0-cabd2005c0a1 service nova] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:07:46.270295 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6f34a90-78ed-437f-96b0-cabd2005c0a1 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:07:46.272156 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-b6f34a90-78ed-437f-96b0-cabd2005c0a1 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:07:46.285691 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6f34a90-78ed-437f-96b0-cabd2005c0a1 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:07:46.287351 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6f34a90-78ed-437f-96b0-cabd2005c0a1 service nova] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:07:46.300551 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1199/2397] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Aug 30 02:07:46 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 251 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 30 02:07:46.360208 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-c0384174-340c-4493-82ca-90c37aa8a6e6 admin admin] The token cache is being invalidate because project e7f4b9bbe0b640688754d04484900cca was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81577) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 30 02:07:46.367277 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-c0384174-340c-4493-82ca-90c37aa8a6e6 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:07:46.367277 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-c0384174-340c-4493-82ca-90c37aa8a6e6 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:07:46.367277 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1199/2398] 10.222.0.27 () {58 vars in 1233 bytes} [Tue Aug 30 02:07:45 2022] DELETE /identity/v3/projects/e7f4b9bbe0b640688754d04484900cca => generated 0 bytes in 488 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 30 02:07:46.368271 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6f34a90-78ed-437f-96b0-cabd2005c0a1 service nova] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:07:46.370152 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba7db5f4-7922-43e2-a412-43477c820ebd None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:07:46.370628 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba7db5f4-7922-43e2-a412-43477c820ebd None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:07:46.371060 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba7db5f4-7922-43e2-a412-43477c820ebd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:07:46.377414 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:07:46.393071 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-ba7db5f4-7922-43e2-a412-43477c820ebd None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:07:46.396052 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-ba7db5f4-7922-43e2-a412-43477c820ebd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:07:46.448562 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6f34a90-78ed-437f-96b0-cabd2005c0a1 service nova] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:07:46.465832 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b584abae-06b8-4ebd-86c6-09db82310b14 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f3ea1580c56f4814a92bbb2acf542905', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4562f3951fbe465280073e60a11e44e2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:07:46.466519 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b584abae-06b8-4ebd-86c6-09db82310b14 service neutron] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:07:46.466799 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b584abae-06b8-4ebd-86c6-09db82310b14 service neutron] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:07:46.467071 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b584abae-06b8-4ebd-86c6-09db82310b14 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:07:46.468892 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-b584abae-06b8-4ebd-86c6-09db82310b14 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:07:46.504235 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b584abae-06b8-4ebd-86c6-09db82310b14 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:07:46.505861 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b584abae-06b8-4ebd-86c6-09db82310b14 service neutron] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:07:46.524583 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1200/2399] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Aug 30 02:07:46 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 157 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 02:07:46.525501 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b584abae-06b8-4ebd-86c6-09db82310b14 service neutron] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:07:46.526863 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-b584abae-06b8-4ebd-86c6-09db82310b14 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:07:46.531655 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1200/2400] 10.222.0.27 () {58 vars in 958 bytes} [Tue Aug 30 02:07:46 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 162 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 02:07:46.540478 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d070bacd-9b21-4802-a56a-8e0a519c52bf None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:07:46.540767 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d070bacd-9b21-4802-a56a-8e0a519c52bf None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:07:46.541257 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d070bacd-9b21-4802-a56a-8e0a519c52bf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:07:46.564949 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-d070bacd-9b21-4802-a56a-8e0a519c52bf None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:07:46.566900 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-d070bacd-9b21-4802-a56a-8e0a519c52bf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:07:46.579306 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1201/2401] 10.222.0.27 () {58 vars in 958 bytes} [Tue Aug 30 02:07:46 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 40 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 02:07:46.586078 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d070bacd-9b21-4802-a56a-8e0a519c52bf None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:07:46.596331 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d070bacd-9b21-4802-a56a-8e0a519c52bf None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:07:46.604454 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddaac95d-fb7a-4968-9b6e-acb9809947c5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:07:46.605716 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddaac95d-fb7a-4968-9b6e-acb9809947c5 admin admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:07:46.608159 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddaac95d-fb7a-4968-9b6e-acb9809947c5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:07:46.608576 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddaac95d-fb7a-4968-9b6e-acb9809947c5 admin admin] PATH_INFO: `/v3/domains` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:07:46.609708 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ddaac95d-fb7a-4968-9b6e-acb9809947c5 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 02:07:46.610165 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ddaac95d-fb7a-4968-9b6e-acb9809947c5 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:07:46.613099 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ddaac95d-fb7a-4968-9b6e-acb9809947c5 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:07:46.629051 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1202/2402] 10.222.0.27 () {58 vars in 1153 bytes} [Tue Aug 30 02:07:46 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 46 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 02:07:46.633703 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddaac95d-fb7a-4968-9b6e-acb9809947c5 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:07:46.639612 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b584abae-06b8-4ebd-86c6-09db82310b14 service neutron] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:07:46.644294 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddaac95d-fb7a-4968-9b6e-acb9809947c5 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:07:46.649066 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-242abe68-68e6-413e-ac30-be3d39d53a52 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '379536ff06fd4f719899e5dde732e63d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4562f3951fbe465280073e60a11e44e2', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:07:46.649873 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8db17fc4-2361-4c8a-a510-cab8f1be6c15 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:07:46.650372 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-242abe68-68e6-413e-ac30-be3d39d53a52 service placement] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:07:46.650754 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-242abe68-68e6-413e-ac30-be3d39d53a52 service placement] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:07:46.650999 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8db17fc4-2361-4c8a-a510-cab8f1be6c15 admin admin] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:07:46.651358 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-242abe68-68e6-413e-ac30-be3d39d53a52 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:07:46.651586 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8db17fc4-2361-4c8a-a510-cab8f1be6c15 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:07:46.652143 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8db17fc4-2361-4c8a-a510-cab8f1be6c15 admin admin] PATH_INFO: `/v3/projects` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:07:46.653031 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8db17fc4-2361-4c8a-a510-cab8f1be6c15 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:07:46.654468 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-242abe68-68e6-413e-ac30-be3d39d53a52 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:07:46.655116 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8db17fc4-2361-4c8a-a510-cab8f1be6c15 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:07:46.678371 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-242abe68-68e6-413e-ac30-be3d39d53a52 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:07:46.680570 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-242abe68-68e6-413e-ac30-be3d39d53a52 service placement] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:07:46.691371 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1201/2403] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Aug 30 02:07:46 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 166 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 02:07:46.748874 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1203/2404] 10.222.0.27 () {60 vars in 1153 bytes} [Tue Aug 30 02:07:46 2022] POST /identity/v3/projects => generated 389 bytes in 115 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 02:07:46.752530 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-242abe68-68e6-413e-ac30-be3d39d53a52 service placement] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:07:46.776526 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-242abe68-68e6-413e-ac30-be3d39d53a52 service placement] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:07:46.843468 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02ec2514-fdd8-40dd-8d89-7aa9c2db5ae7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:07:46.844067 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02ec2514-fdd8-40dd-8d89-7aa9c2db5ae7 admin admin] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:07:46.844427 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02ec2514-fdd8-40dd-8d89-7aa9c2db5ae7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:07:46.844693 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02ec2514-fdd8-40dd-8d89-7aa9c2db5ae7 admin admin] PATH_INFO: `/v3/users` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:07:46.845329 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02ec2514-fdd8-40dd-8d89-7aa9c2db5ae7 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:07:46.847447 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02ec2514-fdd8-40dd-8d89-7aa9c2db5ae7 admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:07:46.938324 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1202/2405] 10.222.0.27 () {60 vars in 1144 bytes} [Tue Aug 30 02:07:46 2022] POST /identity/v3/users => generated 295 bytes in 186 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 02:07:46.942578 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8db17fc4-2361-4c8a-a510-cab8f1be6c15 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:07:46.955807 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8db17fc4-2361-4c8a-a510-cab8f1be6c15 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:07:46.961308 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f91e553c-8172-4a1b-b066-3a124da08a96 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:07:46.961951 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f91e553c-8172-4a1b-b066-3a124da08a96 admin admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:07:46.962231 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f91e553c-8172-4a1b-b066-3a124da08a96 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:07:46.962499 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f91e553c-8172-4a1b-b066-3a124da08a96 admin admin] PATH_INFO: `/v3/roles` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:07:46.963171 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f91e553c-8172-4a1b-b066-3a124da08a96 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:07:46.964843 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f91e553c-8172-4a1b-b066-3a124da08a96 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:07:46.975756 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1204/2406] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Aug 30 02:07:46 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 02:07:46.979880 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02ec2514-fdd8-40dd-8d89-7aa9c2db5ae7 admin admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:07:46.990006 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02ec2514-fdd8-40dd-8d89-7aa9c2db5ae7 admin admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:07:46.995602 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c899412-1b55-40b8-9563-fb986caf6f6f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:07:46.996209 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c899412-1b55-40b8-9563-fb986caf6f6f admin admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:07:46.996485 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c899412-1b55-40b8-9563-fb986caf6f6f admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:07:46.996752 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c899412-1b55-40b8-9563-fb986caf6f6f admin admin] PATH_INFO: `/v3/roles` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:07:46.997386 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c899412-1b55-40b8-9563-fb986caf6f6f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:07:46.999108 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c899412-1b55-40b8-9563-fb986caf6f6f admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:07:47.010146 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1203/2407] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Aug 30 02:07:46 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 02:07:47.014433 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f91e553c-8172-4a1b-b066-3a124da08a96 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:07:47.024447 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f91e553c-8172-4a1b-b066-3a124da08a96 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:07:47.029676 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99885a6b-e209-462c-920a-2c38d8e7282d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:07:47.030344 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99885a6b-e209-462c-920a-2c38d8e7282d admin admin] REQUEST_METHOD: `PUT` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:07:47.030623 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99885a6b-e209-462c-920a-2c38d8e7282d admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:07:47.030905 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99885a6b-e209-462c-920a-2c38d8e7282d admin admin] PATH_INFO: `/v3/projects/abe39137b275484ea6dafbed8d826eb3/users/3eb33b1b787447daa68b651f9686574a/roles/1fb81afcb9f245f3b989add94598a063` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:07:47.048882 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99885a6b-e209-462c-920a-2c38d8e7282d admin admin] RBAC: Authorizing `identity:create_grant(project_id=abe39137b275484ea6dafbed8d826eb3, user_id=3eb33b1b787447daa68b651f9686574a, role_id=1fb81afcb9f245f3b989add94598a063)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:07:47.052095 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99885a6b-e209-462c-920a-2c38d8e7282d admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:07:47.126719 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1205/2408] 10.222.0.27 () {60 vars in 1483 bytes} [Tue Aug 30 02:07:47 2022] PUT /identity/v3/projects/abe39137b275484ea6dafbed8d826eb3/users/3eb33b1b787447daa68b651f9686574a/roles/1fb81afcb9f245f3b989add94598a063 => generated 0 bytes in 113 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 02:07:47.131679 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54cc6ea2-2d82-4773-a23b-7ce151e24722 None None] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:07:47.132494 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54cc6ea2-2d82-4773-a23b-7ce151e24722 None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:07:47.133100 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54cc6ea2-2d82-4773-a23b-7ce151e24722 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:07:47.179363 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.auth.core [None req-54cc6ea2-2d82-4773-a23b-7ce151e24722 None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81578) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:07:47.180633 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-54cc6ea2-2d82-4773-a23b-7ce151e24722 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:07:47.199736 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:07:47.255893 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1204/2409] 10.222.0.27 () {58 vars in 958 bytes} [Tue Aug 30 02:07:47 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 125 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 02:07:47.267921 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99885a6b-e209-462c-920a-2c38d8e7282d admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:07:47.269892 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-99885a6b-e209-462c-920a-2c38d8e7282d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:07:47.356785 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99885a6b-e209-462c-920a-2c38d8e7282d admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:07:47.360671 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6fece91-aebe-459e-baf4-4a759e090a89 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f3ea1580c56f4814a92bbb2acf542905', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4562f3951fbe465280073e60a11e44e2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:07:47.361284 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6fece91-aebe-459e-baf4-4a759e090a89 service neutron] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:07:47.361606 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6fece91-aebe-459e-baf4-4a759e090a89 service neutron] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:07:47.361899 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6fece91-aebe-459e-baf4-4a759e090a89 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:07:47.372644 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6fece91-aebe-459e-baf4-4a759e090a89 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:07:47.374957 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6fece91-aebe-459e-baf4-4a759e090a89 service neutron] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:07:47.385655 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1206/2410] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Aug 30 02:07:47 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 118 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 02:07:49.697168 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54cc6ea2-2d82-4773-a23b-7ce151e24722 None None] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:07:49.699651 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-54cc6ea2-2d82-4773-a23b-7ce151e24722 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:07:49.834069 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54cc6ea2-2d82-4773-a23b-7ce151e24722 None None] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:07:49.850197 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fa19980-0bcf-4c90-8f0d-2b58209f2bc6 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd64f2e71361644589bec8a06e6eceed1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4562f3951fbe465280073e60a11e44e2', 'project_domain_id': 'default', 'roles': ['member', 'service', 'admin', 'ResellerAdmin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:07:49.850954 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fa19980-0bcf-4c90-8f0d-2b58209f2bc6 service nova] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:07:49.851350 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fa19980-0bcf-4c90-8f0d-2b58209f2bc6 service nova] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:07:49.851579 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fa19980-0bcf-4c90-8f0d-2b58209f2bc6 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:07:49.854460 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-1fa19980-0bcf-4c90-8f0d-2b58209f2bc6 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:07:49.995171 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fa19980-0bcf-4c90-8f0d-2b58209f2bc6 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:07:49.997247 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fa19980-0bcf-4c90-8f0d-2b58209f2bc6 service nova] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:07:50.032514 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1205/2411] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Aug 30 02:07:49 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 337 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 02:07:54.820405 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f99df02-c621-432f-8e18-8d4fe805fdfc None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:07:54.821236 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f99df02-c621-432f-8e18-8d4fe805fdfc None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:07:54.821545 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f99df02-c621-432f-8e18-8d4fe805fdfc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:07:54.852077 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-6f99df02-c621-432f-8e18-8d4fe805fdfc None None] MFA Rules not processed for user `3eb33b1b787447daa68b651f9686574a`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:07:54.853513 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-6f99df02-c621-432f-8e18-8d4fe805fdfc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:07:54.857913 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=96,keystone:INSERT=3,keystone:DELETE=1 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:07:54.923595 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1207/2412] 10.222.0.27 () {58 vars in 958 bytes} [Tue Aug 30 02:07:54 2022] POST /identity/v3/auth/tokens => generated 3266 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 02:07:54.939113 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fa19980-0bcf-4c90-8f0d-2b58209f2bc6 service nova] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:07:54.940622 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-1fa19980-0bcf-4c90-8f0d-2b58209f2bc6 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:07:54.958011 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fa19980-0bcf-4c90-8f0d-2b58209f2bc6 service nova] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:07:54.961662 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1046d83b-fa15-446e-9c75-1817c6c272d9 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd64f2e71361644589bec8a06e6eceed1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4562f3951fbe465280073e60a11e44e2', 'project_domain_id': 'default', 'roles': ['member', 'service', 'admin', 'ResellerAdmin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:07:54.962271 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1046d83b-fa15-446e-9c75-1817c6c272d9 service nova] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:07:54.962553 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1046d83b-fa15-446e-9c75-1817c6c272d9 service nova] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:07:54.962829 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1046d83b-fa15-446e-9c75-1817c6c272d9 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:07:54.970654 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1046d83b-fa15-446e-9c75-1817c6c272d9 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:07:54.972208 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1046d83b-fa15-446e-9c75-1817c6c272d9 service nova] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:07:54.982288 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1206/2413] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Aug 30 02:07:54 2022] GET /identity/v3/auth/tokens => generated 3266 bytes in 44 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 02:07:58.270468 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8113cc84-5719-489d-9b71-36f930d518ed None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:07:58.270960 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8113cc84-5719-489d-9b71-36f930d518ed None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:07:58.270960 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8113cc84-5719-489d-9b71-36f930d518ed None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:07:58.291473 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-8113cc84-5719-489d-9b71-36f930d518ed None None] MFA Rules not processed for user `3eb33b1b787447daa68b651f9686574a`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:07:58.292700 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-8113cc84-5719-489d-9b71-36f930d518ed None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:07:58.299660 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1208/2414] 10.222.0.27 () {58 vars in 958 bytes} [Tue Aug 30 02:07:58 2022] POST /identity/v3/auth/tokens => generated 3266 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 02:08:00.636456 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1046d83b-fa15-446e-9c75-1817c6c272d9 service nova] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:08:00.654864 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1046d83b-fa15-446e-9c75-1817c6c272d9 service nova] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:08:00.660419 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48e23663-0c9b-4083-bb3e-fe06b5fb4f32 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd64f2e71361644589bec8a06e6eceed1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4562f3951fbe465280073e60a11e44e2', 'project_domain_id': 'default', 'roles': ['member', 'service', 'admin', 'ResellerAdmin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:08:00.661013 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48e23663-0c9b-4083-bb3e-fe06b5fb4f32 service nova] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:08:00.661373 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48e23663-0c9b-4083-bb3e-fe06b5fb4f32 service nova] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:08:00.661680 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48e23663-0c9b-4083-bb3e-fe06b5fb4f32 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:08:00.663468 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-48e23663-0c9b-4083-bb3e-fe06b5fb4f32 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:08:00.675676 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48e23663-0c9b-4083-bb3e-fe06b5fb4f32 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:08:00.682063 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48e23663-0c9b-4083-bb3e-fe06b5fb4f32 service nova] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:08:00.687851 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1207/2415] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Aug 30 02:08:00 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 52 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 02:08:01.496868 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53484f2c-752d-4f36-8af7-e4bf6fc9fc40 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:08:01.497368 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53484f2c-752d-4f36-8af7-e4bf6fc9fc40 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:08:01.497493 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53484f2c-752d-4f36-8af7-e4bf6fc9fc40 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:08:01.517157 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-53484f2c-752d-4f36-8af7-e4bf6fc9fc40 None None] MFA Rules not processed for user `121c374d49a64335ab74a431cfbc1320`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:08:01.521031 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-53484f2c-752d-4f36-8af7-e4bf6fc9fc40 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:08:01.585698 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1209/2416] 10.222.0.27 () {58 vars in 967 bytes} [Tue Aug 30 02:08:01 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 02:08:01.649321 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48e23663-0c9b-4083-bb3e-fe06b5fb4f32 service nova] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:08:01.649321 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-48e23663-0c9b-4083-bb3e-fe06b5fb4f32 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:08:01.686857 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:08:01.743910 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48e23663-0c9b-4083-bb3e-fe06b5fb4f32 service nova] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:08:01.753151 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-075fc4a4-d71b-4902-970a-c9c73a4f6da0 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b4f0008fde424f22bc1f7307cf422774', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4562f3951fbe465280073e60a11e44e2', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:08:01.757692 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-075fc4a4-d71b-4902-970a-c9c73a4f6da0 service swift] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:08:01.757692 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-075fc4a4-d71b-4902-970a-c9c73a4f6da0 service swift] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:08:01.757692 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-075fc4a4-d71b-4902-970a-c9c73a4f6da0 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:08:01.765701 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-075fc4a4-d71b-4902-970a-c9c73a4f6da0 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:08:01.767301 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-075fc4a4-d71b-4902-970a-c9c73a4f6da0 service swift] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:08:01.776529 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1208/2417] 10.222.0.27 () {60 vars in 1384 bytes} [Tue Aug 30 02:08:01 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 139 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Aug 30 02:08:09.524059 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53484f2c-752d-4f36-8af7-e4bf6fc9fc40 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:08:09.527099 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-53484f2c-752d-4f36-8af7-e4bf6fc9fc40 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:08:09.554701 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53484f2c-752d-4f36-8af7-e4bf6fc9fc40 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:08:09.559691 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7e2582b-71aa-4e67-97c7-ded1fb210b55 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:08:09.560515 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7e2582b-71aa-4e67-97c7-ded1fb210b55 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:08:09.560797 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7e2582b-71aa-4e67-97c7-ded1fb210b55 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:08:09.561071 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7e2582b-71aa-4e67-97c7-ded1fb210b55 admin admin] PATH_INFO: `/v3/users/789377907cfd4f0f9d3123f6418163f7` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:08:09.562255 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7e2582b-71aa-4e67-97c7-ded1fb210b55 admin admin] RBAC: Authorizing `identity:delete_user(user_id=789377907cfd4f0f9d3123f6418163f7)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:08:09.564235 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7e2582b-71aa-4e67-97c7-ded1fb210b55 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:08:09.837590 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-a7e2582b-71aa-4e67-97c7-ded1fb210b55 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '789377907cfd4f0f9d3123f6418163f7'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:08:09.838241 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-a7e2582b-71aa-4e67-97c7-ded1fb210b55 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '789377907cfd4f0f9d3123f6418163f7'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:08:09.838987 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-a7e2582b-71aa-4e67-97c7-ded1fb210b55 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '789377907cfd4f0f9d3123f6418163f7'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:08:09.852252 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-a7e2582b-71aa-4e67-97c7-ded1fb210b55 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '789377907cfd4f0f9d3123f6418163f7'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:08:09.859096 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-a7e2582b-71aa-4e67-97c7-ded1fb210b55 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '789377907cfd4f0f9d3123f6418163f7'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:08:09.997308 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1210/2418] 10.222.0.27 () {58 vars in 1224 bytes} [Tue Aug 30 02:08:09 2022] DELETE /identity/v3/users/789377907cfd4f0f9d3123f6418163f7 => generated 0 bytes in 473 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 02:08:10.368321 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-075fc4a4-d71b-4902-970a-c9c73a4f6da0 service swift] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:08:10.379913 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-075fc4a4-d71b-4902-970a-c9c73a4f6da0 service swift] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:08:10.475739 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14e3ce3d-561f-458d-b9be-36061a28830c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:08:10.477104 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14e3ce3d-561f-458d-b9be-36061a28830c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:08:10.477588 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14e3ce3d-561f-458d-b9be-36061a28830c admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:08:10.478509 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14e3ce3d-561f-458d-b9be-36061a28830c admin admin] PATH_INFO: `/v3/projects/072e2c259a164ecaa45b5ff18fa3ac66` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:08:10.480752 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14e3ce3d-561f-458d-b9be-36061a28830c admin admin] RBAC: Authorizing `identity:delete_project(project_id=072e2c259a164ecaa45b5ff18fa3ac66)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:08:10.488417 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14e3ce3d-561f-458d-b9be-36061a28830c admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:08:10.681568 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-14e3ce3d-561f-458d-b9be-36061a28830c admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '072e2c259a164ecaa45b5ff18fa3ac66', 'request_id': 'req-14e3ce3d-561f-458d-b9be-36061a28830c'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:08:10.689115 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-14e3ce3d-561f-458d-b9be-36061a28830c admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '072e2c259a164ecaa45b5ff18fa3ac66', 'request_id': 'req-14e3ce3d-561f-458d-b9be-36061a28830c'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:08:10.700681 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-14e3ce3d-561f-458d-b9be-36061a28830c admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '072e2c259a164ecaa45b5ff18fa3ac66', 'request_id': 'req-14e3ce3d-561f-458d-b9be-36061a28830c'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:08:10.834247 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-14e3ce3d-561f-458d-b9be-36061a28830c admin admin] The token cache is being invalidate because project 072e2c259a164ecaa45b5ff18fa3ac66 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81578) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 30 02:08:10.842607 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-14e3ce3d-561f-458d-b9be-36061a28830c admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:08:10.842607 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-14e3ce3d-561f-458d-b9be-36061a28830c admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:08:10.842607 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1209/2419] 10.222.0.27 () {58 vars in 1233 bytes} [Tue Aug 30 02:08:10 2022] DELETE /identity/v3/projects/072e2c259a164ecaa45b5ff18fa3ac66 => generated 0 bytes in 469 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 02:08:13.603006 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7e2582b-71aa-4e67-97c7-ded1fb210b55 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:08:13.605118 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-a7e2582b-71aa-4e67-97c7-ded1fb210b55 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:08:13.621292 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7e2582b-71aa-4e67-97c7-ded1fb210b55 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:08:13.633302 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:08:13.678895 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09420018-cefb-426b-a908-aee47d9a5bc8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:08:13.679536 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09420018-cefb-426b-a908-aee47d9a5bc8 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:08:13.679815 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09420018-cefb-426b-a908-aee47d9a5bc8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:08:13.680085 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09420018-cefb-426b-a908-aee47d9a5bc8 admin admin] PATH_INFO: `/v3/users/95c7f09cc2ba4dddae564bbe99ea8a7d` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:08:13.681199 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09420018-cefb-426b-a908-aee47d9a5bc8 admin admin] RBAC: Authorizing `identity:delete_user(user_id=95c7f09cc2ba4dddae564bbe99ea8a7d)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:08:13.683154 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09420018-cefb-426b-a908-aee47d9a5bc8 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:08:13.914988 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-09420018-cefb-426b-a908-aee47d9a5bc8 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '95c7f09cc2ba4dddae564bbe99ea8a7d'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:08:13.915565 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-09420018-cefb-426b-a908-aee47d9a5bc8 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '95c7f09cc2ba4dddae564bbe99ea8a7d'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:08:13.916164 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-09420018-cefb-426b-a908-aee47d9a5bc8 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '95c7f09cc2ba4dddae564bbe99ea8a7d'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:08:13.927716 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-09420018-cefb-426b-a908-aee47d9a5bc8 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '95c7f09cc2ba4dddae564bbe99ea8a7d'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:08:13.935441 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-09420018-cefb-426b-a908-aee47d9a5bc8 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '95c7f09cc2ba4dddae564bbe99ea8a7d'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:08:13.999593 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1211/2420] 10.222.0.27 () {58 vars in 1224 bytes} [Tue Aug 30 02:08:13 2022] DELETE /identity/v3/users/95c7f09cc2ba4dddae564bbe99ea8a7d => generated 0 bytes in 397 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 02:08:14.265287 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14e3ce3d-561f-458d-b9be-36061a28830c admin admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:08:14.266764 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-14e3ce3d-561f-458d-b9be-36061a28830c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:08:14.282237 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14e3ce3d-561f-458d-b9be-36061a28830c admin admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:08:14.334358 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:08:14.344021 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1076853f-9bff-4bd0-9863-9eda047226f5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:08:14.344657 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1076853f-9bff-4bd0-9863-9eda047226f5 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:08:14.344946 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1076853f-9bff-4bd0-9863-9eda047226f5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:08:14.345219 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1076853f-9bff-4bd0-9863-9eda047226f5 admin admin] PATH_INFO: `/v3/projects/b6200be9acd14756908c3fd33adf1506` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:08:14.346559 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1076853f-9bff-4bd0-9863-9eda047226f5 admin admin] RBAC: Authorizing `identity:delete_project(project_id=b6200be9acd14756908c3fd33adf1506)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:08:14.348454 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1076853f-9bff-4bd0-9863-9eda047226f5 admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:08:14.458954 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-1076853f-9bff-4bd0-9863-9eda047226f5 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'b6200be9acd14756908c3fd33adf1506', 'request_id': 'req-1076853f-9bff-4bd0-9863-9eda047226f5'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:08:14.466855 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-1076853f-9bff-4bd0-9863-9eda047226f5 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'b6200be9acd14756908c3fd33adf1506', 'request_id': 'req-1076853f-9bff-4bd0-9863-9eda047226f5'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:08:14.480772 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-1076853f-9bff-4bd0-9863-9eda047226f5 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'b6200be9acd14756908c3fd33adf1506', 'request_id': 'req-1076853f-9bff-4bd0-9863-9eda047226f5'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:08:14.554464 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-1076853f-9bff-4bd0-9863-9eda047226f5 admin admin] The token cache is being invalidate because project b6200be9acd14756908c3fd33adf1506 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81578) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 30 02:08:14.554766 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-1076853f-9bff-4bd0-9863-9eda047226f5 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:08:14.555674 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-1076853f-9bff-4bd0-9863-9eda047226f5 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:08:14.557721 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1210/2421] 10.222.0.27 () {58 vars in 1233 bytes} [Tue Aug 30 02:08:14 2022] DELETE /identity/v3/projects/b6200be9acd14756908c3fd33adf1506 => generated 0 bytes in 293 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 02:08:14.561705 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09420018-cefb-426b-a908-aee47d9a5bc8 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:08:14.563235 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-09420018-cefb-426b-a908-aee47d9a5bc8 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:08:14.576526 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09420018-cefb-426b-a908-aee47d9a5bc8 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:08:14.635433 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57f0399a-50f3-400c-bc8a-a95351e16e28 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:08:14.636206 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57f0399a-50f3-400c-bc8a-a95351e16e28 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:08:14.636342 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57f0399a-50f3-400c-bc8a-a95351e16e28 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:08:14.636618 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57f0399a-50f3-400c-bc8a-a95351e16e28 admin admin] PATH_INFO: `/v3/users/b2927586e099498a91c07560ea4b234c` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:08:14.637770 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57f0399a-50f3-400c-bc8a-a95351e16e28 admin admin] RBAC: Authorizing `identity:delete_user(user_id=b2927586e099498a91c07560ea4b234c)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:08:14.639710 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57f0399a-50f3-400c-bc8a-a95351e16e28 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:08:14.813252 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=84,keystone:DELETE=15,keystone:INSERT=1 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:08:14.822839 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-57f0399a-50f3-400c-bc8a-a95351e16e28 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'b2927586e099498a91c07560ea4b234c'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:08:14.823526 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-57f0399a-50f3-400c-bc8a-a95351e16e28 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'b2927586e099498a91c07560ea4b234c'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:08:14.824122 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-57f0399a-50f3-400c-bc8a-a95351e16e28 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'b2927586e099498a91c07560ea4b234c'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:08:14.837948 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-57f0399a-50f3-400c-bc8a-a95351e16e28 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'b2927586e099498a91c07560ea4b234c'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:08:14.844824 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-57f0399a-50f3-400c-bc8a-a95351e16e28 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'b2927586e099498a91c07560ea4b234c'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:08:14.952841 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1212/2422] 10.222.0.27 () {58 vars in 1224 bytes} [Tue Aug 30 02:08:14 2022] DELETE /identity/v3/users/b2927586e099498a91c07560ea4b234c => generated 0 bytes in 392 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 02:08:15.185671 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1076853f-9bff-4bd0-9863-9eda047226f5 admin admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:08:15.187132 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-1076853f-9bff-4bd0-9863-9eda047226f5 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:08:15.201523 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1076853f-9bff-4bd0-9863-9eda047226f5 admin admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:08:15.264508 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e025098b-2bd6-4dd1-b01a-4fd56bdc1bf2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:08:15.265169 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e025098b-2bd6-4dd1-b01a-4fd56bdc1bf2 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:08:15.265449 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e025098b-2bd6-4dd1-b01a-4fd56bdc1bf2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:08:15.266003 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e025098b-2bd6-4dd1-b01a-4fd56bdc1bf2 admin admin] PATH_INFO: `/v3/projects/28f99120ebae4b29820cc0430ae94f59` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:08:15.266849 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e025098b-2bd6-4dd1-b01a-4fd56bdc1bf2 admin admin] RBAC: Authorizing `identity:delete_project(project_id=28f99120ebae4b29820cc0430ae94f59)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:08:15.268726 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e025098b-2bd6-4dd1-b01a-4fd56bdc1bf2 admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:08:15.410212 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-e025098b-2bd6-4dd1-b01a-4fd56bdc1bf2 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '28f99120ebae4b29820cc0430ae94f59', 'request_id': 'req-e025098b-2bd6-4dd1-b01a-4fd56bdc1bf2'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:08:15.415616 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-e025098b-2bd6-4dd1-b01a-4fd56bdc1bf2 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '28f99120ebae4b29820cc0430ae94f59', 'request_id': 'req-e025098b-2bd6-4dd1-b01a-4fd56bdc1bf2'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:08:15.421700 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-e025098b-2bd6-4dd1-b01a-4fd56bdc1bf2 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '28f99120ebae4b29820cc0430ae94f59', 'request_id': 'req-e025098b-2bd6-4dd1-b01a-4fd56bdc1bf2'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:08:15.509183 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-e025098b-2bd6-4dd1-b01a-4fd56bdc1bf2 admin admin] The token cache is being invalidate because project 28f99120ebae4b29820cc0430ae94f59 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81578) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 30 02:08:15.509492 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-e025098b-2bd6-4dd1-b01a-4fd56bdc1bf2 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:08:15.510126 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-e025098b-2bd6-4dd1-b01a-4fd56bdc1bf2 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:08:15.511635 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1211/2423] 10.222.0.27 () {58 vars in 1233 bytes} [Tue Aug 30 02:08:15 2022] DELETE /identity/v3/projects/28f99120ebae4b29820cc0430ae94f59 => generated 0 bytes in 327 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 02:08:15.516648 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e092d41-f37a-4c0d-9354-33fe96eb3849 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:08:15.516940 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e092d41-f37a-4c0d-9354-33fe96eb3849 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:08:15.517218 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e092d41-f37a-4c0d-9354-33fe96eb3849 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:08:15.537271 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-9e092d41-f37a-4c0d-9354-33fe96eb3849 None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:08:15.539811 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-9e092d41-f37a-4c0d-9354-33fe96eb3849 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:08:15.602040 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1213/2424] 10.222.0.27 () {58 vars in 958 bytes} [Tue Aug 30 02:08:15 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 02:08:15.609846 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40966802-695e-45ea-b390-fbfc9f20b094 None None] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:08:15.610134 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40966802-695e-45ea-b390-fbfc9f20b094 None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:08:15.610411 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40966802-695e-45ea-b390-fbfc9f20b094 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:08:15.631126 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.auth.core [None req-40966802-695e-45ea-b390-fbfc9f20b094 None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81578) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:08:15.632328 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-40966802-695e-45ea-b390-fbfc9f20b094 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:08:15.640438 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1212/2425] 10.222.0.27 () {58 vars in 958 bytes} [Tue Aug 30 02:08:15 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 02:08:15.644693 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e092d41-f37a-4c0d-9354-33fe96eb3849 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:08:15.654810 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e092d41-f37a-4c0d-9354-33fe96eb3849 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:08:15.659434 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63198ff3-5e09-4474-9d9d-a2dc3404b842 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:08:15.660313 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63198ff3-5e09-4474-9d9d-a2dc3404b842 admin admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:08:15.660535 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63198ff3-5e09-4474-9d9d-a2dc3404b842 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:08:15.660837 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63198ff3-5e09-4474-9d9d-a2dc3404b842 admin admin] PATH_INFO: `/v3/domains` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:08:15.661510 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63198ff3-5e09-4474-9d9d-a2dc3404b842 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 02:08:15.661872 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63198ff3-5e09-4474-9d9d-a2dc3404b842 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:08:15.664124 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63198ff3-5e09-4474-9d9d-a2dc3404b842 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:08:15.677974 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1214/2426] 10.222.0.27 () {58 vars in 1153 bytes} [Tue Aug 30 02:08:15 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 02:08:15.682953 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40966802-695e-45ea-b390-fbfc9f20b094 None None] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:08:15.688866 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=84,keystone:DELETE=14,keystone:INSERT=2 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:08:15.692793 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40966802-695e-45ea-b390-fbfc9f20b094 None None] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:08:15.698398 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2757c757-237a-424c-bc07-c1a10fc9ae2a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:08:15.699076 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2757c757-237a-424c-bc07-c1a10fc9ae2a admin admin] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:08:15.699351 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2757c757-237a-424c-bc07-c1a10fc9ae2a admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:08:15.700329 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2757c757-237a-424c-bc07-c1a10fc9ae2a admin admin] PATH_INFO: `/v3/projects` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:08:15.701020 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2757c757-237a-424c-bc07-c1a10fc9ae2a admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:08:15.703050 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2757c757-237a-424c-bc07-c1a10fc9ae2a admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:08:15.843129 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1213/2427] 10.222.0.27 () {60 vars in 1153 bytes} [Tue Aug 30 02:08:15 2022] POST /identity/v3/projects => generated 373 bytes in 161 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 30 02:08:15.848474 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63198ff3-5e09-4474-9d9d-a2dc3404b842 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:08:15.860122 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63198ff3-5e09-4474-9d9d-a2dc3404b842 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:08:15.923396 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b514ea70-9e43-4e15-aee7-d0cb28874036 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:08:15.923925 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b514ea70-9e43-4e15-aee7-d0cb28874036 admin admin] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:08:15.924201 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b514ea70-9e43-4e15-aee7-d0cb28874036 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:08:15.924468 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b514ea70-9e43-4e15-aee7-d0cb28874036 admin admin] PATH_INFO: `/v3/users` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:08:15.925320 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b514ea70-9e43-4e15-aee7-d0cb28874036 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:08:15.927354 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b514ea70-9e43-4e15-aee7-d0cb28874036 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:08:16.017949 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1215/2428] 10.222.0.27 () {60 vars in 1144 bytes} [Tue Aug 30 02:08:15 2022] POST /identity/v3/users => generated 287 bytes in 170 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 30 02:08:16.021766 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2757c757-237a-424c-bc07-c1a10fc9ae2a admin admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:08:16.033133 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2757c757-237a-424c-bc07-c1a10fc9ae2a admin admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:08:16.038328 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-046a0de7-2b4f-4b4a-90ac-71fb69ad60af admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:08:16.039179 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-046a0de7-2b4f-4b4a-90ac-71fb69ad60af admin admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:08:16.039459 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-046a0de7-2b4f-4b4a-90ac-71fb69ad60af admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:08:16.039734 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-046a0de7-2b4f-4b4a-90ac-71fb69ad60af admin admin] PATH_INFO: `/v3/roles` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:08:16.040376 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-046a0de7-2b4f-4b4a-90ac-71fb69ad60af admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:08:16.042105 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-046a0de7-2b4f-4b4a-90ac-71fb69ad60af admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:08:16.052829 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1214/2429] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Aug 30 02:08:16 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 02:08:16.057953 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b514ea70-9e43-4e15-aee7-d0cb28874036 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:08:16.068243 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b514ea70-9e43-4e15-aee7-d0cb28874036 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:08:16.073838 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-913a22d5-c2d0-4bc0-a71b-eaa86ed42100 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:08:16.074585 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-913a22d5-c2d0-4bc0-a71b-eaa86ed42100 admin admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:08:16.074871 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-913a22d5-c2d0-4bc0-a71b-eaa86ed42100 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:08:16.075142 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-913a22d5-c2d0-4bc0-a71b-eaa86ed42100 admin admin] PATH_INFO: `/v3/roles` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:08:16.075817 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-913a22d5-c2d0-4bc0-a71b-eaa86ed42100 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:08:16.077555 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-913a22d5-c2d0-4bc0-a71b-eaa86ed42100 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:08:16.089366 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1216/2430] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Aug 30 02:08:16 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 02:08:16.093748 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-046a0de7-2b4f-4b4a-90ac-71fb69ad60af admin admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:08:16.104755 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-046a0de7-2b4f-4b4a-90ac-71fb69ad60af admin admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:08:16.109640 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c725ba51-6044-4d02-9d9e-64798176b430 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:08:16.110273 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c725ba51-6044-4d02-9d9e-64798176b430 admin admin] REQUEST_METHOD: `PUT` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:08:16.110600 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c725ba51-6044-4d02-9d9e-64798176b430 admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:08:16.110872 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c725ba51-6044-4d02-9d9e-64798176b430 admin admin] PATH_INFO: `/v3/projects/7e5b69be3a9b4ec0b5038f51a918bf27/users/04f1ec1a3d964e34838b14f636cdf13d/roles/1fb81afcb9f245f3b989add94598a063` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:08:16.128036 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c725ba51-6044-4d02-9d9e-64798176b430 admin admin] RBAC: Authorizing `identity:create_grant(project_id=7e5b69be3a9b4ec0b5038f51a918bf27, user_id=04f1ec1a3d964e34838b14f636cdf13d, role_id=1fb81afcb9f245f3b989add94598a063)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:08:16.130702 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c725ba51-6044-4d02-9d9e-64798176b430 admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:08:16.214313 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1215/2431] 10.222.0.27 () {60 vars in 1483 bytes} [Tue Aug 30 02:08:16 2022] PUT /identity/v3/projects/7e5b69be3a9b4ec0b5038f51a918bf27/users/04f1ec1a3d964e34838b14f636cdf13d/roles/1fb81afcb9f245f3b989add94598a063 => generated 0 bytes in 121 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 30 02:08:16.219384 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fc3e2a6-f56b-43eb-af9a-4d3652bbf28e None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:08:16.219672 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fc3e2a6-f56b-43eb-af9a-4d3652bbf28e None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:08:16.219946 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fc3e2a6-f56b-43eb-af9a-4d3652bbf28e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:08:16.246134 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-5fc3e2a6-f56b-43eb-af9a-4d3652bbf28e None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:08:16.247379 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-5fc3e2a6-f56b-43eb-af9a-4d3652bbf28e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:08:16.261705 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=91,keystone:DELETE=5,keystone:INSERT=4 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:08:16.314979 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1217/2432] 10.222.0.27 () {58 vars in 958 bytes} [Tue Aug 30 02:08:16 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 02:08:16.325121 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c725ba51-6044-4d02-9d9e-64798176b430 admin admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:08:16.327739 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-c725ba51-6044-4d02-9d9e-64798176b430 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:08:16.387774 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c725ba51-6044-4d02-9d9e-64798176b430 admin admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:08:16.391659 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b3b47ab-54cf-49ba-b3ed-c476fc961681 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f3ea1580c56f4814a92bbb2acf542905', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4562f3951fbe465280073e60a11e44e2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:08:16.392270 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b3b47ab-54cf-49ba-b3ed-c476fc961681 service neutron] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:08:16.392607 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b3b47ab-54cf-49ba-b3ed-c476fc961681 service neutron] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:08:16.392893 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b3b47ab-54cf-49ba-b3ed-c476fc961681 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:08:16.401907 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b3b47ab-54cf-49ba-b3ed-c476fc961681 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:08:16.403325 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b3b47ab-54cf-49ba-b3ed-c476fc961681 service neutron] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:08:16.412601 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1216/2433] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Aug 30 02:08:16 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 88 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 02:08:22.734101 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80ebdc54-1e6c-461c-84ef-06103c03a76f None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:08:22.734645 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80ebdc54-1e6c-461c-84ef-06103c03a76f None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:08:22.734645 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80ebdc54-1e6c-461c-84ef-06103c03a76f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:08:22.765795 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-80ebdc54-1e6c-461c-84ef-06103c03a76f None None] MFA Rules not processed for user `04f1ec1a3d964e34838b14f636cdf13d`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:08:22.767338 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-80ebdc54-1e6c-461c-84ef-06103c03a76f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:08:22.843718 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1218/2434] 10.222.0.27 () {58 vars in 958 bytes} [Tue Aug 30 02:08:22 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 111 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 02:08:22.848242 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b3b47ab-54cf-49ba-b3ed-c476fc961681 service neutron] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:08:22.859008 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b3b47ab-54cf-49ba-b3ed-c476fc961681 service neutron] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:08:22.863891 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffafcf5d-35fa-47fd-b2ef-454cb49fc180 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:08:22.865962 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffafcf5d-35fa-47fd-b2ef-454cb49fc180 admin admin] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:08:22.866680 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffafcf5d-35fa-47fd-b2ef-454cb49fc180 admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:08:22.867002 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffafcf5d-35fa-47fd-b2ef-454cb49fc180 admin admin] PATH_INFO: `/v3/projects` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:08:22.867585 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffafcf5d-35fa-47fd-b2ef-454cb49fc180 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:08:22.869530 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffafcf5d-35fa-47fd-b2ef-454cb49fc180 admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:08:23.000668 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1217/2435] 10.222.0.27 () {60 vars in 1153 bytes} [Tue Aug 30 02:08:22 2022] POST /identity/v3/projects => generated 371 bytes in 149 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 02:08:23.002919 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80ebdc54-1e6c-461c-84ef-06103c03a76f None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:08:23.018264 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80ebdc54-1e6c-461c-84ef-06103c03a76f None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:08:23.077093 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bd76fc7-bdcb-4c7e-bd24-d0e69354a2ac admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:08:23.077735 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bd76fc7-bdcb-4c7e-bd24-d0e69354a2ac admin admin] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:08:23.078017 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bd76fc7-bdcb-4c7e-bd24-d0e69354a2ac admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:08:23.078286 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bd76fc7-bdcb-4c7e-bd24-d0e69354a2ac admin admin] PATH_INFO: `/v3/users` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:08:23.078910 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7bd76fc7-bdcb-4c7e-bd24-d0e69354a2ac admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:08:23.080820 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7bd76fc7-bdcb-4c7e-bd24-d0e69354a2ac admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:08:23.207779 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1219/2436] 10.222.0.27 () {60 vars in 1144 bytes} [Tue Aug 30 02:08:23 2022] POST /identity/v3/users => generated 292 bytes in 206 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 02:08:23.212026 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffafcf5d-35fa-47fd-b2ef-454cb49fc180 admin admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:08:23.222789 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffafcf5d-35fa-47fd-b2ef-454cb49fc180 admin admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:08:23.228050 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93e7acb1-16f5-4ba7-adc4-3a1066faad5c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:08:23.228657 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93e7acb1-16f5-4ba7-adc4-3a1066faad5c admin admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:08:23.228938 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93e7acb1-16f5-4ba7-adc4-3a1066faad5c admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:08:23.229209 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93e7acb1-16f5-4ba7-adc4-3a1066faad5c admin admin] PATH_INFO: `/v3/roles` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:08:23.229884 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93e7acb1-16f5-4ba7-adc4-3a1066faad5c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:08:23.231475 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93e7acb1-16f5-4ba7-adc4-3a1066faad5c admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:08:23.243637 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1218/2437] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Aug 30 02:08:23 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 02:08:23.247797 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bd76fc7-bdcb-4c7e-bd24-d0e69354a2ac admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:08:23.259031 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bd76fc7-bdcb-4c7e-bd24-d0e69354a2ac admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:08:23.263722 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ead0f8e7-5a33-4195-a399-6df595db1278 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:08:23.264422 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ead0f8e7-5a33-4195-a399-6df595db1278 admin admin] REQUEST_METHOD: `PUT` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:08:23.264654 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ead0f8e7-5a33-4195-a399-6df595db1278 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:08:23.264952 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ead0f8e7-5a33-4195-a399-6df595db1278 admin admin] PATH_INFO: `/v3/projects/77e669ae5c964911b583ab638626d846/users/8d5dd61614e242a3b7c1a28a2627de8c/roles/ead5fac86a0747ea8823a82efcea8ed7` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:08:23.281123 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ead0f8e7-5a33-4195-a399-6df595db1278 admin admin] RBAC: Authorizing `identity:create_grant(project_id=77e669ae5c964911b583ab638626d846, user_id=8d5dd61614e242a3b7c1a28a2627de8c, role_id=ead5fac86a0747ea8823a82efcea8ed7)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:08:23.283810 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ead0f8e7-5a33-4195-a399-6df595db1278 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:08:23.353493 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1220/2438] 10.222.0.27 () {60 vars in 1483 bytes} [Tue Aug 30 02:08:23 2022] PUT /identity/v3/projects/77e669ae5c964911b583ab638626d846/users/8d5dd61614e242a3b7c1a28a2627de8c/roles/ead5fac86a0747ea8823a82efcea8ed7 => generated 0 bytes in 106 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 02:08:23.357439 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93e7acb1-16f5-4ba7-adc4-3a1066faad5c admin admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:08:23.367938 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93e7acb1-16f5-4ba7-adc4-3a1066faad5c admin admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:08:23.428164 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-adb1acac-ef2f-45f1-89ef-79b8c508e820 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:08:23.429388 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adb1acac-ef2f-45f1-89ef-79b8c508e820 admin admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:08:23.429695 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adb1acac-ef2f-45f1-89ef-79b8c508e820 admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:08:23.430191 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adb1acac-ef2f-45f1-89ef-79b8c508e820 admin admin] PATH_INFO: `/v3/roles` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:08:23.430832 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-adb1acac-ef2f-45f1-89ef-79b8c508e820 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:08:23.432404 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-adb1acac-ef2f-45f1-89ef-79b8c508e820 admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:08:23.442509 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1219/2439] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Aug 30 02:08:23 2022] GET /identity/v3/roles => generated 1420 bytes in 86 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 02:08:23.447626 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ead0f8e7-5a33-4195-a399-6df595db1278 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:08:23.457858 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ead0f8e7-5a33-4195-a399-6df595db1278 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:08:23.463247 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac273133-eb6e-47d7-9c9c-dfb851658c96 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:08:23.463888 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac273133-eb6e-47d7-9c9c-dfb851658c96 admin admin] REQUEST_METHOD: `PUT` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:08:23.464166 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac273133-eb6e-47d7-9c9c-dfb851658c96 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:08:23.464437 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac273133-eb6e-47d7-9c9c-dfb851658c96 admin admin] PATH_INFO: `/v3/projects/77e669ae5c964911b583ab638626d846/users/8d5dd61614e242a3b7c1a28a2627de8c/roles/ead5fac86a0747ea8823a82efcea8ed7` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:08:23.466384 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac273133-eb6e-47d7-9c9c-dfb851658c96 admin admin] RBAC: Authorizing `identity:create_grant(project_id=77e669ae5c964911b583ab638626d846, user_id=8d5dd61614e242a3b7c1a28a2627de8c, role_id=ead5fac86a0747ea8823a82efcea8ed7)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:08:23.468999 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac273133-eb6e-47d7-9c9c-dfb851658c96 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:08:23.479852 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1221/2440] 10.222.0.27 () {60 vars in 1483 bytes} [Tue Aug 30 02:08:23 2022] PUT /identity/v3/projects/77e669ae5c964911b583ab638626d846/users/8d5dd61614e242a3b7c1a28a2627de8c/roles/ead5fac86a0747ea8823a82efcea8ed7 => generated 0 bytes in 33 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 02:08:30.644139 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fab9c5d4-399e-4853-84ec-d6c1328c4c32 None None] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:08:30.644654 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fab9c5d4-399e-4853-84ec-d6c1328c4c32 None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:08:30.644863 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fab9c5d4-399e-4853-84ec-d6c1328c4c32 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:08:30.674408 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:08:30.683482 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.auth.core [None req-fab9c5d4-399e-4853-84ec-d6c1328c4c32 None None] MFA Rules not processed for user `8d5dd61614e242a3b7c1a28a2627de8c`. Rule list: `[]` (Enabled: `True`). {{(pid=81578) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:08:30.699440 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-fab9c5d4-399e-4853-84ec-d6c1328c4c32 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:08:30.755382 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1220/2441] 10.222.0.27 () {58 vars in 958 bytes} [Tue Aug 30 02:08:30 2022] POST /identity/v3/auth/tokens => generated 3315 bytes in 112 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 02:08:30.770023 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac273133-eb6e-47d7-9c9c-dfb851658c96 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:08:30.771646 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-ac273133-eb6e-47d7-9c9c-dfb851658c96 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:08:30.777040 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:08:30.850269 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac273133-eb6e-47d7-9c9c-dfb851658c96 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:08:30.856795 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98c8c857-4d33-4199-8f8d-63de0b3c5d27 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd64f2e71361644589bec8a06e6eceed1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4562f3951fbe465280073e60a11e44e2', 'project_domain_id': 'default', 'roles': ['member', 'service', 'admin', 'ResellerAdmin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:08:30.857435 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98c8c857-4d33-4199-8f8d-63de0b3c5d27 service nova] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:08:30.857742 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98c8c857-4d33-4199-8f8d-63de0b3c5d27 service nova] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:08:30.858011 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98c8c857-4d33-4199-8f8d-63de0b3c5d27 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:08:30.915130 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98c8c857-4d33-4199-8f8d-63de0b3c5d27 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:08:30.916730 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98c8c857-4d33-4199-8f8d-63de0b3c5d27 service nova] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:08:30.926292 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1222/2442] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Aug 30 02:08:30 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 157 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 02:08:33.441997 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17f35ba4-300a-4591-bff0-814348b52c28 None None] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:08:33.442511 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17f35ba4-300a-4591-bff0-814348b52c28 None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:08:33.442609 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17f35ba4-300a-4591-bff0-814348b52c28 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:08:33.473419 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.auth.core [None req-17f35ba4-300a-4591-bff0-814348b52c28 None None] MFA Rules not processed for user `04f1ec1a3d964e34838b14f636cdf13d`. Rule list: `[]` (Enabled: `True`). {{(pid=81578) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:08:33.474785 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-17f35ba4-300a-4591-bff0-814348b52c28 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:08:33.501331 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1221/2443] 10.222.0.27 () {58 vars in 958 bytes} [Tue Aug 30 02:08:33 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 60 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 02:08:33.722537 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98c8c857-4d33-4199-8f8d-63de0b3c5d27 service nova] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:08:33.821671 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98c8c857-4d33-4199-8f8d-63de0b3c5d27 service nova] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:08:33.825438 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41d14f89-2ddc-4a56-aef1-2570bc9b14e1 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f3ea1580c56f4814a92bbb2acf542905', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4562f3951fbe465280073e60a11e44e2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:08:33.826209 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41d14f89-2ddc-4a56-aef1-2570bc9b14e1 service neutron] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:08:33.826552 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41d14f89-2ddc-4a56-aef1-2570bc9b14e1 service neutron] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:08:33.826889 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41d14f89-2ddc-4a56-aef1-2570bc9b14e1 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:08:33.828827 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-41d14f89-2ddc-4a56-aef1-2570bc9b14e1 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:08:33.865271 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:08:33.930142 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41d14f89-2ddc-4a56-aef1-2570bc9b14e1 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:08:33.931587 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41d14f89-2ddc-4a56-aef1-2570bc9b14e1 service neutron] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:08:33.952001 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1223/2444] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Aug 30 02:08:33 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 230 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 02:08:37.658206 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbe11fba-fdbe-4bfe-a127-8cd2a1dea236 None None] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:08:37.658754 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbe11fba-fdbe-4bfe-a127-8cd2a1dea236 None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:08:37.658971 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbe11fba-fdbe-4bfe-a127-8cd2a1dea236 None None] PATH_INFO: `/` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:08:37.660463 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1222/2445] 10.222.0.27 () {54 vars in 961 bytes} [Tue Aug 30 02:08:37 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 02:08:37.666169 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-230ded92-8402-489a-99e3-de4639ff7402 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:08:37.666640 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-230ded92-8402-489a-99e3-de4639ff7402 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:08:37.666998 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-230ded92-8402-489a-99e3-de4639ff7402 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:08:37.689675 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-230ded92-8402-489a-99e3-de4639ff7402 None None] MFA Rules not processed for user `379536ff06fd4f719899e5dde732e63d`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:08:37.690930 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-230ded92-8402-489a-99e3-de4639ff7402 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:08:37.784976 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbe11fba-fdbe-4bfe-a127-8cd2a1dea236 None None] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:08:37.786752 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-dbe11fba-fdbe-4bfe-a127-8cd2a1dea236 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:08:37.794970 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1224/2446] 10.222.0.27 () {58 vars in 1058 bytes} [Tue Aug 30 02:08:37 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 130 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 30 02:08:37.809225 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbe11fba-fdbe-4bfe-a127-8cd2a1dea236 None None] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:08:37.811450 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-230ded92-8402-489a-99e3-de4639ff7402 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:08:37.815787 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90fb9353-7fce-4f8b-82cd-d3cd948557bf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:08:37.816527 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90fb9353-7fce-4f8b-82cd-d3cd948557bf admin admin] REQUEST_METHOD: `DELETE` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:08:37.816877 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90fb9353-7fce-4f8b-82cd-d3cd948557bf admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:08:37.817220 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90fb9353-7fce-4f8b-82cd-d3cd948557bf admin admin] PATH_INFO: `/v3/users/bda980735a1f4948ba695e6c2788f3b4` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:08:37.820090 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90fb9353-7fce-4f8b-82cd-d3cd948557bf admin admin] RBAC: Authorizing `identity:delete_user(user_id=bda980735a1f4948ba695e6c2788f3b4)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:08:37.822735 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90fb9353-7fce-4f8b-82cd-d3cd948557bf admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:08:37.829855 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-230ded92-8402-489a-99e3-de4639ff7402 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:08:37.852213 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4770cff8-ea33-49f5-a7ab-de2e049972c3 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '379536ff06fd4f719899e5dde732e63d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4562f3951fbe465280073e60a11e44e2', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:08:37.852213 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4770cff8-ea33-49f5-a7ab-de2e049972c3 service placement] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:08:37.852213 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4770cff8-ea33-49f5-a7ab-de2e049972c3 service placement] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:08:37.852213 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4770cff8-ea33-49f5-a7ab-de2e049972c3 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:08:37.852739 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-4770cff8-ea33-49f5-a7ab-de2e049972c3 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:08:37.864001 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4770cff8-ea33-49f5-a7ab-de2e049972c3 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:08:37.866418 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4770cff8-ea33-49f5-a7ab-de2e049972c3 service placement] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:08:37.879230 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1225/2447] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Aug 30 02:08:37 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 70 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 02:08:38.111619 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-90fb9353-7fce-4f8b-82cd-d3cd948557bf admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'bda980735a1f4948ba695e6c2788f3b4'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:08:38.112917 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-90fb9353-7fce-4f8b-82cd-d3cd948557bf admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'bda980735a1f4948ba695e6c2788f3b4'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:08:38.244145 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-90fb9353-7fce-4f8b-82cd-d3cd948557bf admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'bda980735a1f4948ba695e6c2788f3b4'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:08:38.244791 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-90fb9353-7fce-4f8b-82cd-d3cd948557bf admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'bda980735a1f4948ba695e6c2788f3b4'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:08:38.252617 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-90fb9353-7fce-4f8b-82cd-d3cd948557bf admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'bda980735a1f4948ba695e6c2788f3b4'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:08:38.271708 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1223/2448] 10.222.0.27 () {58 vars in 1224 bytes} [Tue Aug 30 02:08:37 2022] DELETE /identity/v3/users/bda980735a1f4948ba695e6c2788f3b4 => generated 0 bytes in 487 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 02:08:38.516256 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4770cff8-ea33-49f5-a7ab-de2e049972c3 service placement] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:08:38.527394 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4770cff8-ea33-49f5-a7ab-de2e049972c3 service placement] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:08:38.594180 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c844d772-c73e-4862-852c-9b7308d1b948 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:08:38.595003 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c844d772-c73e-4862-852c-9b7308d1b948 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:08:38.595133 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c844d772-c73e-4862-852c-9b7308d1b948 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:08:38.595369 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c844d772-c73e-4862-852c-9b7308d1b948 admin admin] PATH_INFO: `/v3/projects/1cbdcc426c3f4d3090919f1a50e3d626` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:08:38.596527 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c844d772-c73e-4862-852c-9b7308d1b948 admin admin] RBAC: Authorizing `identity:delete_project(project_id=1cbdcc426c3f4d3090919f1a50e3d626)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:08:38.598945 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c844d772-c73e-4862-852c-9b7308d1b948 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:08:38.623069 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:08:38.779643 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-c844d772-c73e-4862-852c-9b7308d1b948 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '1cbdcc426c3f4d3090919f1a50e3d626', 'request_id': 'req-c844d772-c73e-4862-852c-9b7308d1b948'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:08:38.785746 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-c844d772-c73e-4862-852c-9b7308d1b948 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '1cbdcc426c3f4d3090919f1a50e3d626', 'request_id': 'req-c844d772-c73e-4862-852c-9b7308d1b948'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:08:38.793404 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-c844d772-c73e-4862-852c-9b7308d1b948 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '1cbdcc426c3f4d3090919f1a50e3d626', 'request_id': 'req-c844d772-c73e-4862-852c-9b7308d1b948'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:08:38.886824 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-c844d772-c73e-4862-852c-9b7308d1b948 admin admin] The token cache is being invalidate because project 1cbdcc426c3f4d3090919f1a50e3d626 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81577) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 30 02:08:38.887036 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-c844d772-c73e-4862-852c-9b7308d1b948 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:08:38.887652 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-c844d772-c73e-4862-852c-9b7308d1b948 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:08:38.889170 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1226/2449] 10.222.0.27 () {58 vars in 1233 bytes} [Tue Aug 30 02:08:38 2022] DELETE /identity/v3/projects/1cbdcc426c3f4d3090919f1a50e3d626 => generated 0 bytes in 374 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 02:08:38.894350 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-274885e5-35a2-499b-922a-a022c7808646 None None] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:08:38.894917 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-274885e5-35a2-499b-922a-a022c7808646 None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:08:38.894917 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-274885e5-35a2-499b-922a-a022c7808646 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:08:38.924788 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.auth.core [None req-274885e5-35a2-499b-922a-a022c7808646 None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81578) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:08:38.926152 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-274885e5-35a2-499b-922a-a022c7808646 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:08:38.953081 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:08:38.989773 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1224/2450] 10.222.0.27 () {58 vars in 958 bytes} [Tue Aug 30 02:08:38 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 02:08:38.998891 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b432df2-fcc1-495a-a28a-b4fdc8c285bd None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:08:38.999177 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b432df2-fcc1-495a-a28a-b4fdc8c285bd None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:08:38.999448 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b432df2-fcc1-495a-a28a-b4fdc8c285bd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:08:39.019424 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-1b432df2-fcc1-495a-a28a-b4fdc8c285bd None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:08:39.020869 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-1b432df2-fcc1-495a-a28a-b4fdc8c285bd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:08:39.029235 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1227/2451] 10.222.0.27 () {58 vars in 958 bytes} [Tue Aug 30 02:08:38 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 02:08:39.033751 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-274885e5-35a2-499b-922a-a022c7808646 None None] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:08:39.045206 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-274885e5-35a2-499b-922a-a022c7808646 None None] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:08:39.052130 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5155d9b7-77e5-4be5-9463-98eb8f5b6be8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:08:39.052742 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5155d9b7-77e5-4be5-9463-98eb8f5b6be8 admin admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:08:39.053025 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5155d9b7-77e5-4be5-9463-98eb8f5b6be8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:08:39.054235 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5155d9b7-77e5-4be5-9463-98eb8f5b6be8 admin admin] PATH_INFO: `/v3/domains` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:08:39.054686 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5155d9b7-77e5-4be5-9463-98eb8f5b6be8 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81578) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 02:08:39.054989 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5155d9b7-77e5-4be5-9463-98eb8f5b6be8 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:08:39.056606 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5155d9b7-77e5-4be5-9463-98eb8f5b6be8 admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:08:39.068968 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1225/2452] 10.222.0.27 () {58 vars in 1153 bytes} [Tue Aug 30 02:08:39 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 02:08:39.073831 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b432df2-fcc1-495a-a28a-b4fdc8c285bd None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:08:39.086125 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b432df2-fcc1-495a-a28a-b4fdc8c285bd None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:08:39.091042 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-963424f4-6bba-4132-99c7-85053dc9cb7d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:08:39.091652 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-963424f4-6bba-4132-99c7-85053dc9cb7d admin admin] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:08:39.091933 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-963424f4-6bba-4132-99c7-85053dc9cb7d admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:08:39.092205 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-963424f4-6bba-4132-99c7-85053dc9cb7d admin admin] PATH_INFO: `/v3/projects` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:08:39.092846 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-963424f4-6bba-4132-99c7-85053dc9cb7d admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:08:39.094799 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-963424f4-6bba-4132-99c7-85053dc9cb7d admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:08:39.266402 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1228/2453] 10.222.0.27 () {60 vars in 1153 bytes} [Tue Aug 30 02:08:39 2022] POST /identity/v3/projects => generated 383 bytes in 190 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 02:08:39.267571 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5155d9b7-77e5-4be5-9463-98eb8f5b6be8 admin admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:08:39.280676 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5155d9b7-77e5-4be5-9463-98eb8f5b6be8 admin admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:08:39.343183 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f5d94b2-961a-405a-9bc3-0ccbf3838b2d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:08:39.343876 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f5d94b2-961a-405a-9bc3-0ccbf3838b2d admin admin] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:08:39.344219 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f5d94b2-961a-405a-9bc3-0ccbf3838b2d admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:08:39.344582 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f5d94b2-961a-405a-9bc3-0ccbf3838b2d admin admin] PATH_INFO: `/v3/users` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:08:39.345252 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f5d94b2-961a-405a-9bc3-0ccbf3838b2d admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:08:39.347571 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f5d94b2-961a-405a-9bc3-0ccbf3838b2d admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:08:39.452299 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1226/2454] 10.222.0.27 () {60 vars in 1144 bytes} [Tue Aug 30 02:08:39 2022] POST /identity/v3/users => generated 292 bytes in 185 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 02:08:39.457014 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-963424f4-6bba-4132-99c7-85053dc9cb7d admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:08:39.468624 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-963424f4-6bba-4132-99c7-85053dc9cb7d admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:08:39.473577 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9051775-a180-43c3-ab52-feb67d3719ab admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:08:39.474190 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9051775-a180-43c3-ab52-feb67d3719ab admin admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:08:39.478136 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9051775-a180-43c3-ab52-feb67d3719ab admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:08:39.478136 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9051775-a180-43c3-ab52-feb67d3719ab admin admin] PATH_INFO: `/v3/roles` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:08:39.478136 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9051775-a180-43c3-ab52-feb67d3719ab admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:08:39.478136 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9051775-a180-43c3-ab52-feb67d3719ab admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:08:39.490288 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1229/2455] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Aug 30 02:08:39 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 02:08:39.494530 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f5d94b2-961a-405a-9bc3-0ccbf3838b2d admin admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:08:39.506994 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f5d94b2-961a-405a-9bc3-0ccbf3838b2d admin admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:08:39.512453 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea3c3f11-7058-4b10-9fbb-b4506f23e4c0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:08:39.513067 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea3c3f11-7058-4b10-9fbb-b4506f23e4c0 admin admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:08:39.513345 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea3c3f11-7058-4b10-9fbb-b4506f23e4c0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:08:39.513650 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea3c3f11-7058-4b10-9fbb-b4506f23e4c0 admin admin] PATH_INFO: `/v3/roles` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:08:39.514283 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea3c3f11-7058-4b10-9fbb-b4506f23e4c0 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:08:39.515974 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea3c3f11-7058-4b10-9fbb-b4506f23e4c0 admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:08:39.528165 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1227/2456] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Aug 30 02:08:39 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 02:08:39.532563 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9051775-a180-43c3-ab52-feb67d3719ab admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:08:39.542943 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9051775-a180-43c3-ab52-feb67d3719ab admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:08:39.547672 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44dbe23c-2893-4f60-99ec-351ccda9af45 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:08:39.548507 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44dbe23c-2893-4f60-99ec-351ccda9af45 admin admin] REQUEST_METHOD: `PUT` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:08:39.548810 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44dbe23c-2893-4f60-99ec-351ccda9af45 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:08:39.549157 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44dbe23c-2893-4f60-99ec-351ccda9af45 admin admin] PATH_INFO: `/v3/projects/762802bc844445d1bcf4049d1d2dbad4/users/4ad6de12734d41a2a0134a2f624c204d/roles/1fb81afcb9f245f3b989add94598a063` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:08:39.566004 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44dbe23c-2893-4f60-99ec-351ccda9af45 admin admin] RBAC: Authorizing `identity:create_grant(project_id=762802bc844445d1bcf4049d1d2dbad4, user_id=4ad6de12734d41a2a0134a2f624c204d, role_id=1fb81afcb9f245f3b989add94598a063)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:08:39.568658 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44dbe23c-2893-4f60-99ec-351ccda9af45 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:08:39.670367 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1230/2457] 10.222.0.27 () {60 vars in 1483 bytes} [Tue Aug 30 02:08:39 2022] PUT /identity/v3/projects/762802bc844445d1bcf4049d1d2dbad4/users/4ad6de12734d41a2a0134a2f624c204d/roles/1fb81afcb9f245f3b989add94598a063 => generated 0 bytes in 138 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 02:08:39.683752 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5fbcd16-4cd8-41fb-b886-eb29b819d1c1 None None] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:08:39.684042 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5fbcd16-4cd8-41fb-b886-eb29b819d1c1 None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:08:39.684305 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5fbcd16-4cd8-41fb-b886-eb29b819d1c1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:08:39.720239 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.auth.core [None req-f5fbcd16-4cd8-41fb-b886-eb29b819d1c1 None None] MFA Rules not processed for user `4ad6de12734d41a2a0134a2f624c204d`. Rule list: `[]` (Enabled: `True`). {{(pid=81578) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:08:39.720239 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-f5fbcd16-4cd8-41fb-b886-eb29b819d1c1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:08:39.788080 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1228/2458] 10.222.0.27 () {58 vars in 958 bytes} [Tue Aug 30 02:08:39 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 30 02:08:39.799519 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44dbe23c-2893-4f60-99ec-351ccda9af45 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:08:39.801780 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-44dbe23c-2893-4f60-99ec-351ccda9af45 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:08:39.880229 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44dbe23c-2893-4f60-99ec-351ccda9af45 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:08:39.886925 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9403597-47c4-4811-93b6-2fa78b225241 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd64f2e71361644589bec8a06e6eceed1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4562f3951fbe465280073e60a11e44e2', 'project_domain_id': 'default', 'roles': ['member', 'service', 'admin', 'ResellerAdmin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:08:39.887659 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9403597-47c4-4811-93b6-2fa78b225241 service nova] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:08:39.890365 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9403597-47c4-4811-93b6-2fa78b225241 service nova] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:08:39.890365 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9403597-47c4-4811-93b6-2fa78b225241 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:08:39.897790 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9403597-47c4-4811-93b6-2fa78b225241 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:08:39.900170 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9403597-47c4-4811-93b6-2fa78b225241 service nova] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:08:39.909375 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1231/2459] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Aug 30 02:08:39 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 110 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 02:08:49.023128 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5fbcd16-4cd8-41fb-b886-eb29b819d1c1 None None] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:08:49.050881 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:08:49.088734 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5fbcd16-4cd8-41fb-b886-eb29b819d1c1 None None] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:08:49.094195 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0be876e-f5c0-4411-9565-63765f26acbd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:08:49.094817 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0be876e-f5c0-4411-9565-63765f26acbd admin admin] REQUEST_METHOD: `DELETE` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:08:49.095096 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0be876e-f5c0-4411-9565-63765f26acbd admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:08:49.095372 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0be876e-f5c0-4411-9565-63765f26acbd admin admin] PATH_INFO: `/v3/users/4ad6de12734d41a2a0134a2f624c204d` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:08:49.096482 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0be876e-f5c0-4411-9565-63765f26acbd admin admin] RBAC: Authorizing `identity:delete_user(user_id=4ad6de12734d41a2a0134a2f624c204d)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:08:49.098461 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0be876e-f5c0-4411-9565-63765f26acbd admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:08:49.335090 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-a0be876e-f5c0-4411-9565-63765f26acbd admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '4ad6de12734d41a2a0134a2f624c204d'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:08:49.335753 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-a0be876e-f5c0-4411-9565-63765f26acbd admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '4ad6de12734d41a2a0134a2f624c204d'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:08:49.400754 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-a0be876e-f5c0-4411-9565-63765f26acbd admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '4ad6de12734d41a2a0134a2f624c204d'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:08:49.401509 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-a0be876e-f5c0-4411-9565-63765f26acbd admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '4ad6de12734d41a2a0134a2f624c204d'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:08:49.409362 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-a0be876e-f5c0-4411-9565-63765f26acbd admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '4ad6de12734d41a2a0134a2f624c204d'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:08:49.427410 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1229/2460] 10.222.0.27 () {58 vars in 1224 bytes} [Tue Aug 30 02:08:49 2022] DELETE /identity/v3/users/4ad6de12734d41a2a0134a2f624c204d => generated 0 bytes in 405 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 02:08:49.431825 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bc2377a-2528-4ada-bbbe-d9a4d1c3cd90 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:08:49.432342 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bc2377a-2528-4ada-bbbe-d9a4d1c3cd90 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:08:49.432450 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bc2377a-2528-4ada-bbbe-d9a4d1c3cd90 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:08:49.454497 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-9bc2377a-2528-4ada-bbbe-d9a4d1c3cd90 None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:08:49.455707 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-9bc2377a-2528-4ada-bbbe-d9a4d1c3cd90 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:08:49.480377 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:08:49.516653 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1232/2461] 10.222.0.27 () {58 vars in 958 bytes} [Tue Aug 30 02:08:49 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 02:08:49.526565 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0be876e-f5c0-4411-9565-63765f26acbd admin admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:08:49.528240 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-a0be876e-f5c0-4411-9565-63765f26acbd admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:08:49.613309 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0be876e-f5c0-4411-9565-63765f26acbd admin admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:08:49.616969 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92ee2585-3c47-433b-82fd-7b8801b39c00 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f3ea1580c56f4814a92bbb2acf542905', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4562f3951fbe465280073e60a11e44e2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:08:49.617656 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92ee2585-3c47-433b-82fd-7b8801b39c00 service neutron] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:08:49.617941 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92ee2585-3c47-433b-82fd-7b8801b39c00 service neutron] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:08:49.618206 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92ee2585-3c47-433b-82fd-7b8801b39c00 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:08:49.628200 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92ee2585-3c47-433b-82fd-7b8801b39c00 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:08:49.632445 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92ee2585-3c47-433b-82fd-7b8801b39c00 service neutron] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:08:49.639080 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1230/2462] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Aug 30 02:08:49 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 113 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 30 02:08:50.150994 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bc2377a-2528-4ada-bbbe-d9a4d1c3cd90 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:08:50.152424 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-9bc2377a-2528-4ada-bbbe-d9a4d1c3cd90 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:08:50.164688 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bc2377a-2528-4ada-bbbe-d9a4d1c3cd90 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:08:50.171113 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c3f9026-a4f4-494f-9c50-2758440e040c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:08:50.171969 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c3f9026-a4f4-494f-9c50-2758440e040c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:08:50.172330 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c3f9026-a4f4-494f-9c50-2758440e040c admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:08:50.172610 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c3f9026-a4f4-494f-9c50-2758440e040c admin admin] PATH_INFO: `/v3/projects/762802bc844445d1bcf4049d1d2dbad4` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:08:50.174625 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c3f9026-a4f4-494f-9c50-2758440e040c admin admin] RBAC: Authorizing `identity:delete_project(project_id=762802bc844445d1bcf4049d1d2dbad4)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:08:50.176953 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c3f9026-a4f4-494f-9c50-2758440e040c admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:08:50.296377 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-7c3f9026-a4f4-494f-9c50-2758440e040c admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '762802bc844445d1bcf4049d1d2dbad4', 'request_id': 'req-7c3f9026-a4f4-494f-9c50-2758440e040c'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:08:50.304894 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-7c3f9026-a4f4-494f-9c50-2758440e040c admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '762802bc844445d1bcf4049d1d2dbad4', 'request_id': 'req-7c3f9026-a4f4-494f-9c50-2758440e040c'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:08:50.311984 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-7c3f9026-a4f4-494f-9c50-2758440e040c admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '762802bc844445d1bcf4049d1d2dbad4', 'request_id': 'req-7c3f9026-a4f4-494f-9c50-2758440e040c'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:08:50.409554 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-7c3f9026-a4f4-494f-9c50-2758440e040c admin admin] The token cache is being invalidate because project 762802bc844445d1bcf4049d1d2dbad4 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81577) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 30 02:08:50.409812 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-7c3f9026-a4f4-494f-9c50-2758440e040c admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:08:50.410473 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-7c3f9026-a4f4-494f-9c50-2758440e040c admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:08:50.412037 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1233/2463] 10.222.0.27 () {58 vars in 1233 bytes} [Tue Aug 30 02:08:50 2022] DELETE /identity/v3/projects/762802bc844445d1bcf4049d1d2dbad4 => generated 0 bytes in 262 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 02:08:50.418389 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-602a2e30-175d-4e80-9abe-7c4333df8f0b None None] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:08:50.419309 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-602a2e30-175d-4e80-9abe-7c4333df8f0b None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:08:50.419589 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-602a2e30-175d-4e80-9abe-7c4333df8f0b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:08:50.440227 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.auth.core [None req-602a2e30-175d-4e80-9abe-7c4333df8f0b None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81578) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:08:50.441436 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-602a2e30-175d-4e80-9abe-7c4333df8f0b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:08:50.461692 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:08:50.504544 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1231/2464] 10.222.0.27 () {58 vars in 958 bytes} [Tue Aug 30 02:08:50 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 02:08:50.512337 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18935868-8628-4a2f-819a-ac7a716a8ead None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:08:50.512872 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18935868-8628-4a2f-819a-ac7a716a8ead None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:08:50.512872 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18935868-8628-4a2f-819a-ac7a716a8ead None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:08:50.532712 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-18935868-8628-4a2f-819a-ac7a716a8ead None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:08:50.533973 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-18935868-8628-4a2f-819a-ac7a716a8ead None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:08:50.542090 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1234/2465] 10.222.0.27 () {58 vars in 958 bytes} [Tue Aug 30 02:08:50 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 02:08:50.546653 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-602a2e30-175d-4e80-9abe-7c4333df8f0b None None] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:08:50.546823 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18935868-8628-4a2f-819a-ac7a716a8ead None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:08:50.548137 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-602a2e30-175d-4e80-9abe-7c4333df8f0b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:08:50.555830 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18935868-8628-4a2f-819a-ac7a716a8ead None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:08:50.561052 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-116987ca-0a61-42f4-a3ff-6a890d18a232 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:08:50.561678 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-116987ca-0a61-42f4-a3ff-6a890d18a232 admin admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:08:50.561958 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-116987ca-0a61-42f4-a3ff-6a890d18a232 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:08:50.562233 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-116987ca-0a61-42f4-a3ff-6a890d18a232 admin admin] PATH_INFO: `/v3/domains` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:08:50.562878 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-116987ca-0a61-42f4-a3ff-6a890d18a232 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 02:08:50.563183 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-116987ca-0a61-42f4-a3ff-6a890d18a232 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:08:50.564776 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-116987ca-0a61-42f4-a3ff-6a890d18a232 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:08:50.576334 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1235/2466] 10.222.0.27 () {58 vars in 1153 bytes} [Tue Aug 30 02:08:50 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 30 02:08:50.580578 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-116987ca-0a61-42f4-a3ff-6a890d18a232 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:08:50.590153 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-116987ca-0a61-42f4-a3ff-6a890d18a232 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:08:50.594791 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10ac5b42-aa23-413b-84de-bfa2f33ea381 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:08:50.595368 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10ac5b42-aa23-413b-84de-bfa2f33ea381 admin admin] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:08:50.595647 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10ac5b42-aa23-413b-84de-bfa2f33ea381 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:08:50.595921 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10ac5b42-aa23-413b-84de-bfa2f33ea381 admin admin] PATH_INFO: `/v3/projects` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:08:50.596529 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10ac5b42-aa23-413b-84de-bfa2f33ea381 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:08:50.598438 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10ac5b42-aa23-413b-84de-bfa2f33ea381 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:08:50.607921 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-602a2e30-175d-4e80-9abe-7c4333df8f0b None None] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:08:50.611680 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fddabe4-113b-4fb6-8047-0a87cd9968a5 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f3ea1580c56f4814a92bbb2acf542905', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4562f3951fbe465280073e60a11e44e2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:08:50.612293 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fddabe4-113b-4fb6-8047-0a87cd9968a5 service neutron] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:08:50.612578 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fddabe4-113b-4fb6-8047-0a87cd9968a5 service neutron] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:08:50.613608 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fddabe4-113b-4fb6-8047-0a87cd9968a5 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:08:50.615400 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-7fddabe4-113b-4fb6-8047-0a87cd9968a5 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:08:50.626944 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fddabe4-113b-4fb6-8047-0a87cd9968a5 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:08:50.631666 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fddabe4-113b-4fb6-8047-0a87cd9968a5 service neutron] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:08:50.636299 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1232/2467] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Aug 30 02:08:50 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 90 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 02:08:50.726232 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1236/2468] 10.222.0.27 () {60 vars in 1153 bytes} [Tue Aug 30 02:08:50 2022] POST /identity/v3/projects => generated 369 bytes in 135 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 02:08:50.727288 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fddabe4-113b-4fb6-8047-0a87cd9968a5 service neutron] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:08:50.732590 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fddabe4-113b-4fb6-8047-0a87cd9968a5 service neutron] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:08:50.789326 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2b20120-7337-46ef-aa2c-a267fe5eac0f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:08:50.789965 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2b20120-7337-46ef-aa2c-a267fe5eac0f admin admin] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:08:50.790243 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2b20120-7337-46ef-aa2c-a267fe5eac0f admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:08:50.790510 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2b20120-7337-46ef-aa2c-a267fe5eac0f admin admin] PATH_INFO: `/v3/users` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:08:50.791148 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2b20120-7337-46ef-aa2c-a267fe5eac0f admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:08:50.794039 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2b20120-7337-46ef-aa2c-a267fe5eac0f admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:08:50.919448 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1233/2469] 10.222.0.27 () {60 vars in 1144 bytes} [Tue Aug 30 02:08:50 2022] POST /identity/v3/users => generated 285 bytes in 197 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 30 02:08:50.920610 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10ac5b42-aa23-413b-84de-bfa2f33ea381 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:08:50.931483 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10ac5b42-aa23-413b-84de-bfa2f33ea381 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:08:50.936387 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ce596c0-7bda-4a54-bbc2-baac4f0820cb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:08:50.937121 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ce596c0-7bda-4a54-bbc2-baac4f0820cb admin admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:08:50.937404 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ce596c0-7bda-4a54-bbc2-baac4f0820cb admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:08:50.937708 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ce596c0-7bda-4a54-bbc2-baac4f0820cb admin admin] PATH_INFO: `/v3/roles` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:08:50.938348 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ce596c0-7bda-4a54-bbc2-baac4f0820cb admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:08:50.939939 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ce596c0-7bda-4a54-bbc2-baac4f0820cb admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:08:50.950438 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1237/2470] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Aug 30 02:08:50 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 02:08:50.954489 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2b20120-7337-46ef-aa2c-a267fe5eac0f admin admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:08:50.965249 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2b20120-7337-46ef-aa2c-a267fe5eac0f admin admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:08:50.970272 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad667876-b801-4190-a397-4a2f221b4c83 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:08:50.970910 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad667876-b801-4190-a397-4a2f221b4c83 admin admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:08:50.971190 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad667876-b801-4190-a397-4a2f221b4c83 admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:08:50.971462 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad667876-b801-4190-a397-4a2f221b4c83 admin admin] PATH_INFO: `/v3/roles` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:08:50.972122 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad667876-b801-4190-a397-4a2f221b4c83 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:08:50.973736 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad667876-b801-4190-a397-4a2f221b4c83 admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:08:50.986771 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1234/2471] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Aug 30 02:08:50 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 30 02:08:50.991039 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ce596c0-7bda-4a54-bbc2-baac4f0820cb admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:08:51.000998 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ce596c0-7bda-4a54-bbc2-baac4f0820cb admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:08:51.011075 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-323c7019-5b62-4e21-bccc-5ab8e21a5523 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:08:51.011798 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-323c7019-5b62-4e21-bccc-5ab8e21a5523 admin admin] REQUEST_METHOD: `PUT` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:08:51.012080 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-323c7019-5b62-4e21-bccc-5ab8e21a5523 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:08:51.012352 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-323c7019-5b62-4e21-bccc-5ab8e21a5523 admin admin] PATH_INFO: `/v3/projects/f52d4fc5f0664347a361e631c4c9f8a2/users/1e78ac4a539848809f30d0c91262d723/roles/1fb81afcb9f245f3b989add94598a063` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:08:51.028441 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-323c7019-5b62-4e21-bccc-5ab8e21a5523 admin admin] RBAC: Authorizing `identity:create_grant(project_id=f52d4fc5f0664347a361e631c4c9f8a2, user_id=1e78ac4a539848809f30d0c91262d723, role_id=1fb81afcb9f245f3b989add94598a063)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:08:51.031125 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-323c7019-5b62-4e21-bccc-5ab8e21a5523 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:08:51.132222 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1238/2472] 10.222.0.27 () {60 vars in 1483 bytes} [Tue Aug 30 02:08:50 2022] PUT /identity/v3/projects/f52d4fc5f0664347a361e631c4c9f8a2/users/1e78ac4a539848809f30d0c91262d723/roles/1fb81afcb9f245f3b989add94598a063 => generated 0 bytes in 142 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 02:08:51.148364 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca7d249c-8c24-4e39-97ce-6b2cb9f17b30 None None] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:08:51.148655 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca7d249c-8c24-4e39-97ce-6b2cb9f17b30 None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:08:51.148933 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca7d249c-8c24-4e39-97ce-6b2cb9f17b30 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:08:51.179926 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.auth.core [None req-ca7d249c-8c24-4e39-97ce-6b2cb9f17b30 None None] MFA Rules not processed for user `1e78ac4a539848809f30d0c91262d723`. Rule list: `[]` (Enabled: `True`). {{(pid=81578) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:08:51.181241 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-ca7d249c-8c24-4e39-97ce-6b2cb9f17b30 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:08:51.222624 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:08:51.256446 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1235/2473] 10.222.0.27 () {58 vars in 958 bytes} [Tue Aug 30 02:08:51 2022] POST /identity/v3/auth/tokens => generated 3246 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 02:08:51.260804 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-323c7019-5b62-4e21-bccc-5ab8e21a5523 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:08:51.304360 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:08:51.328052 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-323c7019-5b62-4e21-bccc-5ab8e21a5523 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:08:51.332806 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07770f71-0b83-48a2-b3b7-4ed2e06407c8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:08:51.333410 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07770f71-0b83-48a2-b3b7-4ed2e06407c8 admin admin] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:08:51.333734 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07770f71-0b83-48a2-b3b7-4ed2e06407c8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:08:51.334007 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07770f71-0b83-48a2-b3b7-4ed2e06407c8 admin admin] PATH_INFO: `/v3/projects` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:08:51.334630 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07770f71-0b83-48a2-b3b7-4ed2e06407c8 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:08:51.336613 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07770f71-0b83-48a2-b3b7-4ed2e06407c8 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:08:51.458163 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1239/2474] 10.222.0.27 () {60 vars in 1153 bytes} [Tue Aug 30 02:08:51 2022] POST /identity/v3/projects => generated 369 bytes in 198 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 02:08:51.469232 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca7d249c-8c24-4e39-97ce-6b2cb9f17b30 None None] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:08:51.474338 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca7d249c-8c24-4e39-97ce-6b2cb9f17b30 None None] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:08:51.546332 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2f613f7-9ec8-416d-9985-c757f854b3c1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:08:51.547111 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2f613f7-9ec8-416d-9985-c757f854b3c1 admin admin] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:08:51.547457 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2f613f7-9ec8-416d-9985-c757f854b3c1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:08:51.547913 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2f613f7-9ec8-416d-9985-c757f854b3c1 admin admin] PATH_INFO: `/v3/users` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:08:51.548626 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2f613f7-9ec8-416d-9985-c757f854b3c1 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:08:51.550796 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2f613f7-9ec8-416d-9985-c757f854b3c1 admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:08:51.641345 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1236/2475] 10.222.0.27 () {60 vars in 1144 bytes} [Tue Aug 30 02:08:51 2022] POST /identity/v3/users => generated 291 bytes in 182 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 30 02:08:51.645079 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07770f71-0b83-48a2-b3b7-4ed2e06407c8 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:08:51.655837 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07770f71-0b83-48a2-b3b7-4ed2e06407c8 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:08:51.660506 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0ffa30a-2e24-45e0-b573-a83869f6c179 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:08:51.661121 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0ffa30a-2e24-45e0-b573-a83869f6c179 admin admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:08:51.665875 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0ffa30a-2e24-45e0-b573-a83869f6c179 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:08:51.665875 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0ffa30a-2e24-45e0-b573-a83869f6c179 admin admin] PATH_INFO: `/v3/roles` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:08:51.665875 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0ffa30a-2e24-45e0-b573-a83869f6c179 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:08:51.665875 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0ffa30a-2e24-45e0-b573-a83869f6c179 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:08:51.674920 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1240/2476] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Aug 30 02:08:51 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 30 02:08:51.678926 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2f613f7-9ec8-416d-9985-c757f854b3c1 admin admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:08:51.690166 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2f613f7-9ec8-416d-9985-c757f854b3c1 admin admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:08:51.694975 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5f58372-4c23-48a6-b366-d529f920f127 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:08:51.695928 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5f58372-4c23-48a6-b366-d529f920f127 admin admin] REQUEST_METHOD: `PUT` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:08:51.696274 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5f58372-4c23-48a6-b366-d529f920f127 admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:08:51.696615 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5f58372-4c23-48a6-b366-d529f920f127 admin admin] PATH_INFO: `/v3/projects/9e8d3354c0f74f37b56e1bd4dbe6788f/users/163ccd2c9d4648119bde055b479c3f8d/roles/ead5fac86a0747ea8823a82efcea8ed7` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:08:51.720482 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5f58372-4c23-48a6-b366-d529f920f127 admin admin] RBAC: Authorizing `identity:create_grant(project_id=9e8d3354c0f74f37b56e1bd4dbe6788f, user_id=163ccd2c9d4648119bde055b479c3f8d, role_id=ead5fac86a0747ea8823a82efcea8ed7)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:08:51.723400 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5f58372-4c23-48a6-b366-d529f920f127 admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:08:51.810570 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1237/2477] 10.222.0.27 () {60 vars in 1483 bytes} [Tue Aug 30 02:08:51 2022] PUT /identity/v3/projects/9e8d3354c0f74f37b56e1bd4dbe6788f/users/163ccd2c9d4648119bde055b479c3f8d/roles/ead5fac86a0747ea8823a82efcea8ed7 => generated 0 bytes in 132 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 02:08:51.814649 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0ffa30a-2e24-45e0-b573-a83869f6c179 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:08:51.825406 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0ffa30a-2e24-45e0-b573-a83869f6c179 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:08:51.895653 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-412345a9-152b-470c-b7a6-12b189203e0f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:08:51.901690 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-412345a9-152b-470c-b7a6-12b189203e0f admin admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:08:51.901690 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-412345a9-152b-470c-b7a6-12b189203e0f admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:08:51.901690 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-412345a9-152b-470c-b7a6-12b189203e0f admin admin] PATH_INFO: `/v3/roles` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:08:51.901690 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-412345a9-152b-470c-b7a6-12b189203e0f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:08:51.901690 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-412345a9-152b-470c-b7a6-12b189203e0f admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:08:51.910501 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1241/2478] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Aug 30 02:08:51 2022] GET /identity/v3/roles => generated 1420 bytes in 96 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 30 02:08:51.914747 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5f58372-4c23-48a6-b366-d529f920f127 admin admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:08:51.925288 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5f58372-4c23-48a6-b366-d529f920f127 admin admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:08:51.930446 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6491213f-081d-4590-9a37-93a0af8246ed admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:08:51.931107 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6491213f-081d-4590-9a37-93a0af8246ed admin admin] REQUEST_METHOD: `PUT` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:08:51.931393 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6491213f-081d-4590-9a37-93a0af8246ed admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:08:51.931658 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6491213f-081d-4590-9a37-93a0af8246ed admin admin] PATH_INFO: `/v3/projects/9e8d3354c0f74f37b56e1bd4dbe6788f/users/163ccd2c9d4648119bde055b479c3f8d/roles/ead5fac86a0747ea8823a82efcea8ed7` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:08:51.933627 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6491213f-081d-4590-9a37-93a0af8246ed admin admin] RBAC: Authorizing `identity:create_grant(project_id=9e8d3354c0f74f37b56e1bd4dbe6788f, user_id=163ccd2c9d4648119bde055b479c3f8d, role_id=ead5fac86a0747ea8823a82efcea8ed7)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:08:51.936246 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6491213f-081d-4590-9a37-93a0af8246ed admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:08:51.946829 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1238/2479] 10.222.0.27 () {60 vars in 1483 bytes} [Tue Aug 30 02:08:51 2022] PUT /identity/v3/projects/9e8d3354c0f74f37b56e1bd4dbe6788f/users/163ccd2c9d4648119bde055b479c3f8d/roles/ead5fac86a0747ea8823a82efcea8ed7 => generated 0 bytes in 33 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 02:08:51.958966 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f6f8093-5663-461f-a4c2-25597f913fa1 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:08:51.965546 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f6f8093-5663-461f-a4c2-25597f913fa1 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:08:51.965546 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f6f8093-5663-461f-a4c2-25597f913fa1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:08:51.992495 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-6f6f8093-5663-461f-a4c2-25597f913fa1 None None] MFA Rules not processed for user `163ccd2c9d4648119bde055b479c3f8d`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:08:51.993849 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-6f6f8093-5663-461f-a4c2-25597f913fa1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:08:52.065080 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1242/2480] 10.222.0.27 () {58 vars in 958 bytes} [Tue Aug 30 02:08:51 2022] POST /identity/v3/auth/tokens => generated 3313 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 02:08:52.076859 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6491213f-081d-4590-9a37-93a0af8246ed admin admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:08:52.078593 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-6491213f-081d-4590-9a37-93a0af8246ed admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:08:52.166009 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6491213f-081d-4590-9a37-93a0af8246ed admin admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:08:52.170251 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ead8d6f-db3f-4bd2-800d-967f25b79d47 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ab936b3767a8417aa80c8c8f1b84b667', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4562f3951fbe465280073e60a11e44e2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:08:52.171003 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ead8d6f-db3f-4bd2-800d-967f25b79d47 service cinder] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:08:52.171234 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ead8d6f-db3f-4bd2-800d-967f25b79d47 service cinder] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:08:52.171513 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ead8d6f-db3f-4bd2-800d-967f25b79d47 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:08:52.181158 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ead8d6f-db3f-4bd2-800d-967f25b79d47 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:08:52.182726 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ead8d6f-db3f-4bd2-800d-967f25b79d47 service cinder] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:08:52.193288 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1239/2481] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Aug 30 02:08:52 2022] GET /identity/v3/auth/tokens => generated 3313 bytes in 117 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 02:08:52.332553 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f6f8093-5663-461f-a4c2-25597f913fa1 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:08:52.334019 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-6f6f8093-5663-461f-a4c2-25597f913fa1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:08:52.356958 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f6f8093-5663-461f-a4c2-25597f913fa1 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:08:52.360999 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cce4dca-9eca-4705-b692-296cc06daac6 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ab936b3767a8417aa80c8c8f1b84b667', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4562f3951fbe465280073e60a11e44e2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:08:52.361666 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cce4dca-9eca-4705-b692-296cc06daac6 service cinder] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:08:52.361953 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cce4dca-9eca-4705-b692-296cc06daac6 service cinder] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:08:52.362226 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cce4dca-9eca-4705-b692-296cc06daac6 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:08:52.402811 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:08:52.431942 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5cce4dca-9eca-4705-b692-296cc06daac6 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:08:52.434041 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5cce4dca-9eca-4705-b692-296cc06daac6 service cinder] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:08:52.443938 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1243/2482] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Aug 30 02:08:52 2022] GET /identity/v3/auth/tokens => generated 3246 bytes in 112 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 02:08:57.932892 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ead8d6f-db3f-4bd2-800d-967f25b79d47 service cinder] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:08:57.946498 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:08:57.995263 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ead8d6f-db3f-4bd2-800d-967f25b79d47 service cinder] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:08:58.000793 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e5b94d6-4504-497b-a6b8-6ecc024c8759 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:08:58.001439 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e5b94d6-4504-497b-a6b8-6ecc024c8759 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:08:58.001754 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e5b94d6-4504-497b-a6b8-6ecc024c8759 admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:08:58.002013 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e5b94d6-4504-497b-a6b8-6ecc024c8759 admin admin] PATH_INFO: `/v3/users/1e78ac4a539848809f30d0c91262d723` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:08:58.003104 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e5b94d6-4504-497b-a6b8-6ecc024c8759 admin admin] RBAC: Authorizing `identity:delete_user(user_id=1e78ac4a539848809f30d0c91262d723)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:08:58.005002 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e5b94d6-4504-497b-a6b8-6ecc024c8759 admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:08:58.197889 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-8e5b94d6-4504-497b-a6b8-6ecc024c8759 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '1e78ac4a539848809f30d0c91262d723'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:08:58.198547 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-8e5b94d6-4504-497b-a6b8-6ecc024c8759 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '1e78ac4a539848809f30d0c91262d723'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:08:58.358845 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-8e5b94d6-4504-497b-a6b8-6ecc024c8759 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '1e78ac4a539848809f30d0c91262d723'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:08:58.362330 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-8e5b94d6-4504-497b-a6b8-6ecc024c8759 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '1e78ac4a539848809f30d0c91262d723'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:08:58.373792 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-8e5b94d6-4504-497b-a6b8-6ecc024c8759 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '1e78ac4a539848809f30d0c91262d723'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:08:58.394244 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1240/2483] 10.222.0.27 () {58 vars in 1224 bytes} [Tue Aug 30 02:08:57 2022] DELETE /identity/v3/users/1e78ac4a539848809f30d0c91262d723 => generated 0 bytes in 462 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 02:08:58.398963 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5762d97b-6101-4fd8-a580-b373884b691a None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:08:58.399966 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5762d97b-6101-4fd8-a580-b373884b691a None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:08:58.399966 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5762d97b-6101-4fd8-a580-b373884b691a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:08:58.423330 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-5762d97b-6101-4fd8-a580-b373884b691a None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:08:58.423507 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-5762d97b-6101-4fd8-a580-b373884b691a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:08:58.509676 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1244/2484] 10.222.0.27 () {58 vars in 958 bytes} [Tue Aug 30 02:08:58 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 112 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 02:08:58.519020 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e5b94d6-4504-497b-a6b8-6ecc024c8759 admin admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:08:58.521276 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-8e5b94d6-4504-497b-a6b8-6ecc024c8759 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:08:58.596953 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e5b94d6-4504-497b-a6b8-6ecc024c8759 admin admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:08:58.600984 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23c1d0fd-c8fc-49e9-ab96-f591e6cd6183 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f3ea1580c56f4814a92bbb2acf542905', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4562f3951fbe465280073e60a11e44e2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:08:58.602225 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23c1d0fd-c8fc-49e9-ab96-f591e6cd6183 service neutron] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:08:58.602514 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23c1d0fd-c8fc-49e9-ab96-f591e6cd6183 service neutron] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:08:58.602806 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23c1d0fd-c8fc-49e9-ab96-f591e6cd6183 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:08:58.617246 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23c1d0fd-c8fc-49e9-ab96-f591e6cd6183 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:08:58.618713 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23c1d0fd-c8fc-49e9-ab96-f591e6cd6183 service neutron] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:08:58.632019 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1241/2485] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Aug 30 02:08:58 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 113 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 02:08:59.117290 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5762d97b-6101-4fd8-a580-b373884b691a None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:08:59.119311 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-5762d97b-6101-4fd8-a580-b373884b691a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:08:59.135869 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5762d97b-6101-4fd8-a580-b373884b691a None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:08:59.141186 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e49bcc6e-3767-4e06-89cd-00308bcd9004 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:08:59.142092 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e49bcc6e-3767-4e06-89cd-00308bcd9004 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:08:59.142372 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e49bcc6e-3767-4e06-89cd-00308bcd9004 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:08:59.142645 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e49bcc6e-3767-4e06-89cd-00308bcd9004 admin admin] PATH_INFO: `/v3/projects/f52d4fc5f0664347a361e631c4c9f8a2` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:08:59.144332 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e49bcc6e-3767-4e06-89cd-00308bcd9004 admin admin] RBAC: Authorizing `identity:delete_project(project_id=f52d4fc5f0664347a361e631c4c9f8a2)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:08:59.148750 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e49bcc6e-3767-4e06-89cd-00308bcd9004 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:08:59.346236 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-e49bcc6e-3767-4e06-89cd-00308bcd9004 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'f52d4fc5f0664347a361e631c4c9f8a2', 'request_id': 'req-e49bcc6e-3767-4e06-89cd-00308bcd9004'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:08:59.354024 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-e49bcc6e-3767-4e06-89cd-00308bcd9004 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'f52d4fc5f0664347a361e631c4c9f8a2', 'request_id': 'req-e49bcc6e-3767-4e06-89cd-00308bcd9004'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:08:59.362764 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-e49bcc6e-3767-4e06-89cd-00308bcd9004 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'f52d4fc5f0664347a361e631c4c9f8a2', 'request_id': 'req-e49bcc6e-3767-4e06-89cd-00308bcd9004'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:08:59.472607 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-e49bcc6e-3767-4e06-89cd-00308bcd9004 admin admin] The token cache is being invalidate because project f52d4fc5f0664347a361e631c4c9f8a2 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81577) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 30 02:08:59.473339 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-e49bcc6e-3767-4e06-89cd-00308bcd9004 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:08:59.473489 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-e49bcc6e-3767-4e06-89cd-00308bcd9004 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:08:59.474996 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1245/2486] 10.222.0.27 () {58 vars in 1233 bytes} [Tue Aug 30 02:08:59 2022] DELETE /identity/v3/projects/f52d4fc5f0664347a361e631c4c9f8a2 => generated 0 bytes in 358 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 30 02:08:59.479403 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23c1d0fd-c8fc-49e9-ab96-f591e6cd6183 service neutron] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:08:59.490517 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23c1d0fd-c8fc-49e9-ab96-f591e6cd6183 service neutron] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:08:59.528564 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:08:59.554333 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-814f8ce3-978c-42d0-b39d-63f5f7b7dec4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:08:59.554970 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-814f8ce3-978c-42d0-b39d-63f5f7b7dec4 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:08:59.555250 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-814f8ce3-978c-42d0-b39d-63f5f7b7dec4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:08:59.555521 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-814f8ce3-978c-42d0-b39d-63f5f7b7dec4 admin admin] PATH_INFO: `/v3/users/163ccd2c9d4648119bde055b479c3f8d` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:08:59.556644 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-814f8ce3-978c-42d0-b39d-63f5f7b7dec4 admin admin] RBAC: Authorizing `identity:delete_user(user_id=163ccd2c9d4648119bde055b479c3f8d)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:08:59.560954 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-814f8ce3-978c-42d0-b39d-63f5f7b7dec4 admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:08:59.737025 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-814f8ce3-978c-42d0-b39d-63f5f7b7dec4 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '163ccd2c9d4648119bde055b479c3f8d'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:08:59.737758 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-814f8ce3-978c-42d0-b39d-63f5f7b7dec4 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '163ccd2c9d4648119bde055b479c3f8d'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:08:59.793701 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-814f8ce3-978c-42d0-b39d-63f5f7b7dec4 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '163ccd2c9d4648119bde055b479c3f8d'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:08:59.794323 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-814f8ce3-978c-42d0-b39d-63f5f7b7dec4 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '163ccd2c9d4648119bde055b479c3f8d'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:08:59.802582 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-814f8ce3-978c-42d0-b39d-63f5f7b7dec4 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '163ccd2c9d4648119bde055b479c3f8d'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:08:59.820825 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1242/2487] 10.222.0.27 () {58 vars in 1224 bytes} [Tue Aug 30 02:08:59 2022] DELETE /identity/v3/users/163ccd2c9d4648119bde055b479c3f8d => generated 0 bytes in 342 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 02:08:59.928828 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e49bcc6e-3767-4e06-89cd-00308bcd9004 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:08:59.930386 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-e49bcc6e-3767-4e06-89cd-00308bcd9004 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:08:59.995391 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e49bcc6e-3767-4e06-89cd-00308bcd9004 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:09:00.000585 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:09:00.053310 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b02dedd2-a7e4-4d95-90b4-4726c0d12b26 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f3ea1580c56f4814a92bbb2acf542905', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4562f3951fbe465280073e60a11e44e2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:09:00.054082 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b02dedd2-a7e4-4d95-90b4-4726c0d12b26 service neutron] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:09:00.054447 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b02dedd2-a7e4-4d95-90b4-4726c0d12b26 service neutron] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:09:00.054718 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b02dedd2-a7e4-4d95-90b4-4726c0d12b26 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:09:00.057112 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-b02dedd2-a7e4-4d95-90b4-4726c0d12b26 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:09:00.126343 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b02dedd2-a7e4-4d95-90b4-4726c0d12b26 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:09:00.127780 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b02dedd2-a7e4-4d95-90b4-4726c0d12b26 service neutron] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:09:00.138492 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1246/2488] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Aug 30 02:08:59 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 210 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 02:09:00.259596 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-814f8ce3-978c-42d0-b39d-63f5f7b7dec4 admin admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:09:00.261277 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-814f8ce3-978c-42d0-b39d-63f5f7b7dec4 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:09:00.277994 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-814f8ce3-978c-42d0-b39d-63f5f7b7dec4 admin admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:09:00.285374 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8885591a-32ba-4223-84b7-fdc59bd43df4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:09:00.285901 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8885591a-32ba-4223-84b7-fdc59bd43df4 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:09:00.286343 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8885591a-32ba-4223-84b7-fdc59bd43df4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:09:00.286709 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8885591a-32ba-4223-84b7-fdc59bd43df4 admin admin] PATH_INFO: `/v3/projects/9e8d3354c0f74f37b56e1bd4dbe6788f` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:09:00.287858 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8885591a-32ba-4223-84b7-fdc59bd43df4 admin admin] RBAC: Authorizing `identity:delete_project(project_id=9e8d3354c0f74f37b56e1bd4dbe6788f)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:09:00.290471 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8885591a-32ba-4223-84b7-fdc59bd43df4 admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:09:00.402764 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-8885591a-32ba-4223-84b7-fdc59bd43df4 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '9e8d3354c0f74f37b56e1bd4dbe6788f', 'request_id': 'req-8885591a-32ba-4223-84b7-fdc59bd43df4'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:09:00.410112 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-8885591a-32ba-4223-84b7-fdc59bd43df4 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '9e8d3354c0f74f37b56e1bd4dbe6788f', 'request_id': 'req-8885591a-32ba-4223-84b7-fdc59bd43df4'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:09:00.417302 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-8885591a-32ba-4223-84b7-fdc59bd43df4 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '9e8d3354c0f74f37b56e1bd4dbe6788f', 'request_id': 'req-8885591a-32ba-4223-84b7-fdc59bd43df4'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:09:00.529402 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-8885591a-32ba-4223-84b7-fdc59bd43df4 admin admin] The token cache is being invalidate because project 9e8d3354c0f74f37b56e1bd4dbe6788f was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81578) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 30 02:09:00.530187 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-8885591a-32ba-4223-84b7-fdc59bd43df4 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:09:00.530733 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-8885591a-32ba-4223-84b7-fdc59bd43df4 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:09:00.532239 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1243/2489] 10.222.0.27 () {58 vars in 1233 bytes} [Tue Aug 30 02:09:00 2022] DELETE /identity/v3/projects/9e8d3354c0f74f37b56e1bd4dbe6788f => generated 0 bytes in 273 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 02:09:00.537593 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e880d9a-6c18-4f8e-a627-eaed1e8e661d None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:09:00.538042 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e880d9a-6c18-4f8e-a627-eaed1e8e661d None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:09:00.538427 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e880d9a-6c18-4f8e-a627-eaed1e8e661d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:09:00.559231 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-6e880d9a-6c18-4f8e-a627-eaed1e8e661d None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:09:00.563758 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-6e880d9a-6c18-4f8e-a627-eaed1e8e661d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:09:00.639360 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1247/2490] 10.222.0.27 () {58 vars in 958 bytes} [Tue Aug 30 02:09:00 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 02:09:00.647552 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-286547fd-b71a-4023-ac07-8eaa8536cd20 None None] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:09:00.647842 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-286547fd-b71a-4023-ac07-8eaa8536cd20 None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:09:00.648118 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-286547fd-b71a-4023-ac07-8eaa8536cd20 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:09:00.669751 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.auth.core [None req-286547fd-b71a-4023-ac07-8eaa8536cd20 None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81578) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:09:00.671057 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-286547fd-b71a-4023-ac07-8eaa8536cd20 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:09:00.679269 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1244/2491] 10.222.0.27 () {58 vars in 958 bytes} [Tue Aug 30 02:09:00 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 02:09:00.688010 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e880d9a-6c18-4f8e-a627-eaed1e8e661d None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:09:00.699605 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e880d9a-6c18-4f8e-a627-eaed1e8e661d None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:09:00.704783 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02a59777-5939-4ca0-a2c1-3fe12b4411dd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:09:00.705436 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02a59777-5939-4ca0-a2c1-3fe12b4411dd admin admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:09:00.705750 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02a59777-5939-4ca0-a2c1-3fe12b4411dd admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:09:00.706023 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02a59777-5939-4ca0-a2c1-3fe12b4411dd admin admin] PATH_INFO: `/v3/domains` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:09:00.706686 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02a59777-5939-4ca0-a2c1-3fe12b4411dd admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 02:09:00.706988 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02a59777-5939-4ca0-a2c1-3fe12b4411dd admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:09:00.708850 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02a59777-5939-4ca0-a2c1-3fe12b4411dd admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:09:00.723345 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1248/2492] 10.222.0.27 () {58 vars in 1153 bytes} [Tue Aug 30 02:09:00 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 02:09:00.728352 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-286547fd-b71a-4023-ac07-8eaa8536cd20 None None] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:09:00.738104 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-286547fd-b71a-4023-ac07-8eaa8536cd20 None None] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:09:00.757815 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2d4d9f5-3db0-44e9-81be-3e8dac4b25b3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:09:00.758880 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2d4d9f5-3db0-44e9-81be-3e8dac4b25b3 admin admin] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:09:00.759381 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2d4d9f5-3db0-44e9-81be-3e8dac4b25b3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:09:00.762133 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2d4d9f5-3db0-44e9-81be-3e8dac4b25b3 admin admin] PATH_INFO: `/v3/projects` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:09:00.762133 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2d4d9f5-3db0-44e9-81be-3e8dac4b25b3 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:09:00.765609 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2d4d9f5-3db0-44e9-81be-3e8dac4b25b3 admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:09:00.919505 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1245/2493] 10.222.0.27 () {60 vars in 1153 bytes} [Tue Aug 30 02:09:00 2022] POST /identity/v3/projects => generated 385 bytes in 186 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 02:09:00.920816 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02a59777-5939-4ca0-a2c1-3fe12b4411dd admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:09:00.931826 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02a59777-5939-4ca0-a2c1-3fe12b4411dd admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:09:00.983509 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:09:00.997403 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40a8ab93-3ae7-4993-a115-fe4c18810e7a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:09:01.002474 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40a8ab93-3ae7-4993-a115-fe4c18810e7a admin admin] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:09:01.002474 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40a8ab93-3ae7-4993-a115-fe4c18810e7a admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:09:01.002474 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40a8ab93-3ae7-4993-a115-fe4c18810e7a admin admin] PATH_INFO: `/v3/users` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:09:01.002474 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40a8ab93-3ae7-4993-a115-fe4c18810e7a admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:09:01.002474 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40a8ab93-3ae7-4993-a115-fe4c18810e7a admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:09:01.122536 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1249/2494] 10.222.0.27 () {60 vars in 1144 bytes} [Tue Aug 30 02:09:00 2022] POST /identity/v3/users => generated 293 bytes in 202 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 30 02:09:01.126712 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2d4d9f5-3db0-44e9-81be-3e8dac4b25b3 admin admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:09:01.138090 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2d4d9f5-3db0-44e9-81be-3e8dac4b25b3 admin admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:09:01.143719 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-337f74c6-e6f4-4427-b9c8-3e19651d5309 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:09:01.144427 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-337f74c6-e6f4-4427-b9c8-3e19651d5309 admin admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:09:01.144777 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-337f74c6-e6f4-4427-b9c8-3e19651d5309 admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:09:01.145134 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-337f74c6-e6f4-4427-b9c8-3e19651d5309 admin admin] PATH_INFO: `/v3/roles` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:09:01.146103 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-337f74c6-e6f4-4427-b9c8-3e19651d5309 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:09:01.148211 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-337f74c6-e6f4-4427-b9c8-3e19651d5309 admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:09:01.155795 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=80,keystone:DELETE=17,keystone:INSERT=3 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:09:01.167082 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1246/2495] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Aug 30 02:09:01 2022] GET /identity/v3/roles => generated 1420 bytes in 41 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 02:09:01.171961 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40a8ab93-3ae7-4993-a115-fe4c18810e7a admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:09:01.184411 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40a8ab93-3ae7-4993-a115-fe4c18810e7a admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:09:01.205429 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cceb347-2dd0-40dc-8d03-8bb798a7e9b5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:09:01.206359 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cceb347-2dd0-40dc-8d03-8bb798a7e9b5 admin admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:09:01.206766 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cceb347-2dd0-40dc-8d03-8bb798a7e9b5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:09:01.207138 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cceb347-2dd0-40dc-8d03-8bb798a7e9b5 admin admin] PATH_INFO: `/v3/roles` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:09:01.207921 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2cceb347-2dd0-40dc-8d03-8bb798a7e9b5 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:09:01.210198 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2cceb347-2dd0-40dc-8d03-8bb798a7e9b5 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:09:01.226570 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1250/2496] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Aug 30 02:09:01 2022] GET /identity/v3/roles => generated 1420 bytes in 55 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 02:09:01.231989 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-337f74c6-e6f4-4427-b9c8-3e19651d5309 admin admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:09:01.251924 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-337f74c6-e6f4-4427-b9c8-3e19651d5309 admin admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:09:01.259992 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f3d2237-ebd3-4cfd-9e66-a61ab98f9d59 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:09:01.260685 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f3d2237-ebd3-4cfd-9e66-a61ab98f9d59 admin admin] REQUEST_METHOD: `PUT` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:09:01.261341 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f3d2237-ebd3-4cfd-9e66-a61ab98f9d59 admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:09:01.263686 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f3d2237-ebd3-4cfd-9e66-a61ab98f9d59 admin admin] PATH_INFO: `/v3/projects/3e1069cdcd5b4c4c93a792ba86928e79/users/39e161dfd48b49d783fa39ec2dc599d7/roles/1fb81afcb9f245f3b989add94598a063` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:09:01.295194 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f3d2237-ebd3-4cfd-9e66-a61ab98f9d59 admin admin] RBAC: Authorizing `identity:create_grant(project_id=3e1069cdcd5b4c4c93a792ba86928e79, user_id=39e161dfd48b49d783fa39ec2dc599d7, role_id=1fb81afcb9f245f3b989add94598a063)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:09:01.298121 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f3d2237-ebd3-4cfd-9e66-a61ab98f9d59 admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:09:01.441901 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1247/2497] 10.222.0.27 () {60 vars in 1483 bytes} [Tue Aug 30 02:09:01 2022] PUT /identity/v3/projects/3e1069cdcd5b4c4c93a792ba86928e79/users/39e161dfd48b49d783fa39ec2dc599d7/roles/1fb81afcb9f245f3b989add94598a063 => generated 0 bytes in 211 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 02:09:01.452320 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82b7d874-1a01-427f-aab1-f590cf9acde7 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:09:01.452607 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82b7d874-1a01-427f-aab1-f590cf9acde7 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:09:01.452882 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82b7d874-1a01-427f-aab1-f590cf9acde7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:09:01.482350 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-82b7d874-1a01-427f-aab1-f590cf9acde7 None None] MFA Rules not processed for user `39e161dfd48b49d783fa39ec2dc599d7`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:09:01.483586 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-82b7d874-1a01-427f-aab1-f590cf9acde7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:09:01.549525 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1251/2498] 10.222.0.27 () {58 vars in 958 bytes} [Tue Aug 30 02:09:01 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 02:09:01.553807 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f3d2237-ebd3-4cfd-9e66-a61ab98f9d59 admin admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:09:01.619682 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f3d2237-ebd3-4cfd-9e66-a61ab98f9d59 admin admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:09:01.624781 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-436ac2e1-72b6-4d89-8f3d-95a3ff108f2d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:09:01.625385 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-436ac2e1-72b6-4d89-8f3d-95a3ff108f2d admin admin] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:09:01.625700 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-436ac2e1-72b6-4d89-8f3d-95a3ff108f2d admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:09:01.625974 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-436ac2e1-72b6-4d89-8f3d-95a3ff108f2d admin admin] PATH_INFO: `/v3/projects` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:09:01.626617 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-436ac2e1-72b6-4d89-8f3d-95a3ff108f2d admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:09:01.628526 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-436ac2e1-72b6-4d89-8f3d-95a3ff108f2d admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:09:01.740036 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1248/2499] 10.222.0.27 () {60 vars in 1153 bytes} [Tue Aug 30 02:09:01 2022] POST /identity/v3/projects => generated 385 bytes in 187 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 02:09:01.744047 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82b7d874-1a01-427f-aab1-f590cf9acde7 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:09:01.757024 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82b7d874-1a01-427f-aab1-f590cf9acde7 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:09:01.832170 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fef27b82-a588-4b3b-93ea-8384fe09b788 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:09:01.832932 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fef27b82-a588-4b3b-93ea-8384fe09b788 admin admin] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:09:01.833219 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fef27b82-a588-4b3b-93ea-8384fe09b788 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:09:01.834564 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fef27b82-a588-4b3b-93ea-8384fe09b788 admin admin] PATH_INFO: `/v3/users` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:09:01.834564 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fef27b82-a588-4b3b-93ea-8384fe09b788 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:09:01.838278 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fef27b82-a588-4b3b-93ea-8384fe09b788 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:09:01.910570 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1252/2500] 10.222.0.27 () {60 vars in 1144 bytes} [Tue Aug 30 02:09:01 2022] POST /identity/v3/users => generated 293 bytes in 167 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 02:09:01.914606 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-436ac2e1-72b6-4d89-8f3d-95a3ff108f2d admin admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:09:01.926609 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-436ac2e1-72b6-4d89-8f3d-95a3ff108f2d admin admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:09:01.933068 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89d21d02-5f3a-4e62-978f-5878fb1f2935 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:09:01.933564 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89d21d02-5f3a-4e62-978f-5878fb1f2935 admin admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:09:01.933854 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89d21d02-5f3a-4e62-978f-5878fb1f2935 admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:09:01.934128 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89d21d02-5f3a-4e62-978f-5878fb1f2935 admin admin] PATH_INFO: `/v3/roles` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:09:01.934803 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89d21d02-5f3a-4e62-978f-5878fb1f2935 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:09:01.936365 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89d21d02-5f3a-4e62-978f-5878fb1f2935 admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:09:01.948460 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1249/2501] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Aug 30 02:09:01 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 02:09:01.953404 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fef27b82-a588-4b3b-93ea-8384fe09b788 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:09:01.964275 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fef27b82-a588-4b3b-93ea-8384fe09b788 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:09:01.972892 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8a3ac3e-f373-4799-a92e-019e5fe18104 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:09:01.973908 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8a3ac3e-f373-4799-a92e-019e5fe18104 admin admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:09:01.974267 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8a3ac3e-f373-4799-a92e-019e5fe18104 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:09:01.974610 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8a3ac3e-f373-4799-a92e-019e5fe18104 admin admin] PATH_INFO: `/v3/roles` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:09:01.975377 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8a3ac3e-f373-4799-a92e-019e5fe18104 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:09:01.977878 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8a3ac3e-f373-4799-a92e-019e5fe18104 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:09:01.990478 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1253/2502] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Aug 30 02:09:01 2022] GET /identity/v3/roles => generated 1420 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 02:09:01.995055 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89d21d02-5f3a-4e62-978f-5878fb1f2935 admin admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:09:02.005560 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89d21d02-5f3a-4e62-978f-5878fb1f2935 admin admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:09:02.012395 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-382b3165-7b89-4d4e-8b92-6081a2daaf9d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:09:02.015939 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-382b3165-7b89-4d4e-8b92-6081a2daaf9d admin admin] REQUEST_METHOD: `PUT` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:09:02.015939 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-382b3165-7b89-4d4e-8b92-6081a2daaf9d admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:09:02.015939 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-382b3165-7b89-4d4e-8b92-6081a2daaf9d admin admin] PATH_INFO: `/v3/projects/5df0340c865e422c8c9b8b6aeaa4a9a7/users/c9f80f8047a14665afd2de282bad7ed3/roles/1fb81afcb9f245f3b989add94598a063` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:09:02.031081 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-382b3165-7b89-4d4e-8b92-6081a2daaf9d admin admin] RBAC: Authorizing `identity:create_grant(project_id=5df0340c865e422c8c9b8b6aeaa4a9a7, user_id=c9f80f8047a14665afd2de282bad7ed3, role_id=1fb81afcb9f245f3b989add94598a063)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:09:02.034351 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-382b3165-7b89-4d4e-8b92-6081a2daaf9d admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:09:02.105228 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1250/2503] 10.222.0.27 () {60 vars in 1483 bytes} [Tue Aug 30 02:09:01 2022] PUT /identity/v3/projects/5df0340c865e422c8c9b8b6aeaa4a9a7/users/c9f80f8047a14665afd2de282bad7ed3/roles/1fb81afcb9f245f3b989add94598a063 => generated 0 bytes in 111 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 30 02:09:02.117509 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d0e52da-c06a-473f-9c86-61feabb4288e None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:09:02.117792 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d0e52da-c06a-473f-9c86-61feabb4288e None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:09:02.118069 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d0e52da-c06a-473f-9c86-61feabb4288e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:09:02.166755 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-3d0e52da-c06a-473f-9c86-61feabb4288e None None] MFA Rules not processed for user `c9f80f8047a14665afd2de282bad7ed3`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:09:02.167265 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-3d0e52da-c06a-473f-9c86-61feabb4288e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:09:02.201087 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=94,keystone:INSERT=6 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:09:02.257615 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1254/2504] 10.222.0.27 () {58 vars in 958 bytes} [Tue Aug 30 02:09:02 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 141 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 02:09:02.263026 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-382b3165-7b89-4d4e-8b92-6081a2daaf9d admin admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:09:02.362754 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-382b3165-7b89-4d4e-8b92-6081a2daaf9d admin admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:09:02.366299 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ec2731d-082c-40b2-929d-a1b011753067 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:09:02.366958 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ec2731d-082c-40b2-929d-a1b011753067 admin admin] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:09:02.367237 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ec2731d-082c-40b2-929d-a1b011753067 admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:09:02.367504 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ec2731d-082c-40b2-929d-a1b011753067 admin admin] PATH_INFO: `/v3/projects` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:09:02.368141 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ec2731d-082c-40b2-929d-a1b011753067 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:09:02.370317 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ec2731d-082c-40b2-929d-a1b011753067 admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:09:02.477637 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1251/2505] 10.222.0.27 () {60 vars in 1153 bytes} [Tue Aug 30 02:09:02 2022] POST /identity/v3/projects => generated 385 bytes in 215 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 02:09:02.482713 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d0e52da-c06a-473f-9c86-61feabb4288e None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:09:02.493877 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d0e52da-c06a-473f-9c86-61feabb4288e None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:09:02.555176 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18a2e81c-76f6-441e-ad89-70086d3ac775 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:09:02.555865 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18a2e81c-76f6-441e-ad89-70086d3ac775 admin admin] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:09:02.556063 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18a2e81c-76f6-441e-ad89-70086d3ac775 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:09:02.556336 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18a2e81c-76f6-441e-ad89-70086d3ac775 admin admin] PATH_INFO: `/v3/users` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:09:02.556969 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18a2e81c-76f6-441e-ad89-70086d3ac775 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:09:02.559431 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18a2e81c-76f6-441e-ad89-70086d3ac775 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:09:02.664229 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1255/2506] 10.222.0.27 () {60 vars in 1144 bytes} [Tue Aug 30 02:09:02 2022] POST /identity/v3/users => generated 299 bytes in 182 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 30 02:09:02.669527 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ec2731d-082c-40b2-929d-a1b011753067 admin admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:09:02.680522 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ec2731d-082c-40b2-929d-a1b011753067 admin admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:09:02.689698 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-875d4e65-dec0-4a0d-96fd-a419e3212322 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:09:02.690816 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-875d4e65-dec0-4a0d-96fd-a419e3212322 admin admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:09:02.691097 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-875d4e65-dec0-4a0d-96fd-a419e3212322 admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:09:02.692850 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-875d4e65-dec0-4a0d-96fd-a419e3212322 admin admin] PATH_INFO: `/v3/roles` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:09:02.692850 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-875d4e65-dec0-4a0d-96fd-a419e3212322 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:09:02.698479 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-875d4e65-dec0-4a0d-96fd-a419e3212322 admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:09:02.702230 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:09:02.714152 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1252/2507] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Aug 30 02:09:02 2022] GET /identity/v3/roles => generated 1420 bytes in 46 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 02:09:02.719334 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18a2e81c-76f6-441e-ad89-70086d3ac775 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:09:02.734366 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18a2e81c-76f6-441e-ad89-70086d3ac775 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:09:02.739269 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6ebc8fb-dd36-437c-aea2-e8d20af02a3a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:09:02.739969 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6ebc8fb-dd36-437c-aea2-e8d20af02a3a admin admin] REQUEST_METHOD: `PUT` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:09:02.740256 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6ebc8fb-dd36-437c-aea2-e8d20af02a3a admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:09:02.740531 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6ebc8fb-dd36-437c-aea2-e8d20af02a3a admin admin] PATH_INFO: `/v3/projects/58b53773ef1b45c3a467233747c4f75e/users/0da22da0af7449aeb838f9219c8add65/roles/ead5fac86a0747ea8823a82efcea8ed7` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:09:02.757714 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6ebc8fb-dd36-437c-aea2-e8d20af02a3a admin admin] RBAC: Authorizing `identity:create_grant(project_id=58b53773ef1b45c3a467233747c4f75e, user_id=0da22da0af7449aeb838f9219c8add65, role_id=ead5fac86a0747ea8823a82efcea8ed7)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:09:02.760485 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6ebc8fb-dd36-437c-aea2-e8d20af02a3a admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:09:02.847954 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1256/2508] 10.222.0.27 () {60 vars in 1483 bytes} [Tue Aug 30 02:09:02 2022] PUT /identity/v3/projects/58b53773ef1b45c3a467233747c4f75e/users/0da22da0af7449aeb838f9219c8add65/roles/ead5fac86a0747ea8823a82efcea8ed7 => generated 0 bytes in 129 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 02:09:02.851855 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-875d4e65-dec0-4a0d-96fd-a419e3212322 admin admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:09:02.864287 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-875d4e65-dec0-4a0d-96fd-a419e3212322 admin admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:09:02.927706 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe0e0dbf-d9d5-4b88-aff3-e75d2f428a79 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:09:02.928314 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe0e0dbf-d9d5-4b88-aff3-e75d2f428a79 admin admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:09:02.928604 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe0e0dbf-d9d5-4b88-aff3-e75d2f428a79 admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:09:02.928878 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe0e0dbf-d9d5-4b88-aff3-e75d2f428a79 admin admin] PATH_INFO: `/v3/roles` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:09:02.929558 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe0e0dbf-d9d5-4b88-aff3-e75d2f428a79 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:09:02.931155 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe0e0dbf-d9d5-4b88-aff3-e75d2f428a79 admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:09:02.952111 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1253/2509] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Aug 30 02:09:02 2022] GET /identity/v3/roles => generated 1420 bytes in 101 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 02:09:02.956393 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6ebc8fb-dd36-437c-aea2-e8d20af02a3a admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:09:02.972784 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6ebc8fb-dd36-437c-aea2-e8d20af02a3a admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:09:02.978444 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce81f49b-7182-4b68-93f6-89f6a3b6208f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:09:02.979496 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce81f49b-7182-4b68-93f6-89f6a3b6208f admin admin] REQUEST_METHOD: `PUT` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:09:02.979624 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce81f49b-7182-4b68-93f6-89f6a3b6208f admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:09:02.980039 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce81f49b-7182-4b68-93f6-89f6a3b6208f admin admin] PATH_INFO: `/v3/projects/58b53773ef1b45c3a467233747c4f75e/users/0da22da0af7449aeb838f9219c8add65/roles/ead5fac86a0747ea8823a82efcea8ed7` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:09:02.987676 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce81f49b-7182-4b68-93f6-89f6a3b6208f admin admin] RBAC: Authorizing `identity:create_grant(project_id=58b53773ef1b45c3a467233747c4f75e, user_id=0da22da0af7449aeb838f9219c8add65, role_id=ead5fac86a0747ea8823a82efcea8ed7)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:09:02.987676 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce81f49b-7182-4b68-93f6-89f6a3b6208f admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:09:02.997320 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1257/2510] 10.222.0.27 () {60 vars in 1483 bytes} [Tue Aug 30 02:09:02 2022] PUT /identity/v3/projects/58b53773ef1b45c3a467233747c4f75e/users/0da22da0af7449aeb838f9219c8add65/roles/ead5fac86a0747ea8823a82efcea8ed7 => generated 0 bytes in 42 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 02:09:03.007790 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35a92b47-2da0-4721-a12e-42c58a61443c None None] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:09:03.008309 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35a92b47-2da0-4721-a12e-42c58a61443c None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:09:03.008650 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35a92b47-2da0-4721-a12e-42c58a61443c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:09:03.039227 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.auth.core [None req-35a92b47-2da0-4721-a12e-42c58a61443c None None] MFA Rules not processed for user `0da22da0af7449aeb838f9219c8add65`. Rule list: `[]` (Enabled: `True`). {{(pid=81578) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:09:03.040375 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-35a92b47-2da0-4721-a12e-42c58a61443c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:09:03.127539 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1254/2511] 10.222.0.27 () {58 vars in 958 bytes} [Tue Aug 30 02:09:03 2022] POST /identity/v3/auth/tokens => generated 3329 bytes in 121 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 02:09:03.139615 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce81f49b-7182-4b68-93f6-89f6a3b6208f admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:09:03.141357 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-ce81f49b-7182-4b68-93f6-89f6a3b6208f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:09:03.220457 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce81f49b-7182-4b68-93f6-89f6a3b6208f admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:09:03.226255 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f176582-aa58-40ff-b48f-bf3a419d811d service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ab936b3767a8417aa80c8c8f1b84b667', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4562f3951fbe465280073e60a11e44e2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:09:03.227288 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f176582-aa58-40ff-b48f-bf3a419d811d service cinder] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:09:03.240036 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f176582-aa58-40ff-b48f-bf3a419d811d service cinder] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:09:03.240265 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f176582-aa58-40ff-b48f-bf3a419d811d service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:09:03.266093 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f176582-aa58-40ff-b48f-bf3a419d811d service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:09:03.267678 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f176582-aa58-40ff-b48f-bf3a419d811d service cinder] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:09:03.280333 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1258/2512] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Aug 30 02:09:03 2022] GET /identity/v3/auth/tokens => generated 3329 bytes in 141 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 02:09:04.178296 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35a92b47-2da0-4721-a12e-42c58a61443c None None] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:09:04.181708 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-35a92b47-2da0-4721-a12e-42c58a61443c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:09:04.251064 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35a92b47-2da0-4721-a12e-42c58a61443c None None] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:09:04.256177 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e04be288-143a-4fd0-bd54-8ec2a2d81185 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '379536ff06fd4f719899e5dde732e63d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4562f3951fbe465280073e60a11e44e2', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:09:04.256890 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e04be288-143a-4fd0-bd54-8ec2a2d81185 service placement] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:09:04.257167 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e04be288-143a-4fd0-bd54-8ec2a2d81185 service placement] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:09:04.257435 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e04be288-143a-4fd0-bd54-8ec2a2d81185 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:09:04.259282 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-e04be288-143a-4fd0-bd54-8ec2a2d81185 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:09:04.271592 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e04be288-143a-4fd0-bd54-8ec2a2d81185 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:09:04.273320 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e04be288-143a-4fd0-bd54-8ec2a2d81185 service placement] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:09:04.284184 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1255/2513] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Aug 30 02:09:04 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 106 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 02:09:04.334512 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f176582-aa58-40ff-b48f-bf3a419d811d service cinder] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:09:04.335917 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-0f176582-aa58-40ff-b48f-bf3a419d811d service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:09:04.359276 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:09:04.404872 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f176582-aa58-40ff-b48f-bf3a419d811d service cinder] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:09:04.409443 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebd5c873-588c-4bbd-b3a8-d656410aaafb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:09:04.410101 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebd5c873-588c-4bbd-b3a8-d656410aaafb admin admin] REQUEST_METHOD: `DELETE` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:09:04.410383 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebd5c873-588c-4bbd-b3a8-d656410aaafb admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:09:04.410655 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebd5c873-588c-4bbd-b3a8-d656410aaafb admin admin] PATH_INFO: `/v3/users/7af1c1123b2e458b84248865287bd00e` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:09:04.411777 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebd5c873-588c-4bbd-b3a8-d656410aaafb admin admin] RBAC: Authorizing `identity:delete_user(user_id=7af1c1123b2e458b84248865287bd00e)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:09:04.413710 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebd5c873-588c-4bbd-b3a8-d656410aaafb admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:09:04.566811 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-ebd5c873-588c-4bbd-b3a8-d656410aaafb admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '7af1c1123b2e458b84248865287bd00e'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:09:04.567819 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-ebd5c873-588c-4bbd-b3a8-d656410aaafb admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '7af1c1123b2e458b84248865287bd00e'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:09:04.568629 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-ebd5c873-588c-4bbd-b3a8-d656410aaafb admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '7af1c1123b2e458b84248865287bd00e'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:09:04.581737 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e04be288-143a-4fd0-bd54-8ec2a2d81185 service placement] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:09:04.583323 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-ebd5c873-588c-4bbd-b3a8-d656410aaafb admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '7af1c1123b2e458b84248865287bd00e'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:09:04.583522 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-e04be288-143a-4fd0-bd54-8ec2a2d81185 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:09:04.591325 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-ebd5c873-588c-4bbd-b3a8-d656410aaafb admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '7af1c1123b2e458b84248865287bd00e'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:09:04.596660 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e04be288-143a-4fd0-bd54-8ec2a2d81185 service placement] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:09:04.600325 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3c70705-a448-4e33-ae0c-b85e551169cd service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ab936b3767a8417aa80c8c8f1b84b667', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4562f3951fbe465280073e60a11e44e2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:09:04.600914 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3c70705-a448-4e33-ae0c-b85e551169cd service cinder] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:09:04.601192 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3c70705-a448-4e33-ae0c-b85e551169cd service cinder] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:09:04.601605 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3c70705-a448-4e33-ae0c-b85e551169cd service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:09:04.603605 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-c3c70705-a448-4e33-ae0c-b85e551169cd service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:09:04.610953 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:09:04.684651 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1259/2514] 10.222.0.27 () {58 vars in 1224 bytes} [Tue Aug 30 02:09:04 2022] DELETE /identity/v3/users/7af1c1123b2e458b84248865287bd00e => generated 0 bytes in 351 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 02:09:04.684864 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3c70705-a448-4e33-ae0c-b85e551169cd service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:09:04.686575 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3c70705-a448-4e33-ae0c-b85e551169cd service cinder] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:09:04.698499 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1256/2515] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Aug 30 02:09:04 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 117 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 30 02:09:04.989094 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebd5c873-588c-4bbd-b3a8-d656410aaafb admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:09:04.990699 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-ebd5c873-588c-4bbd-b3a8-d656410aaafb admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:09:05.061683 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebd5c873-588c-4bbd-b3a8-d656410aaafb admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:09:05.067755 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84d211b6-2fb3-4a51-a715-e6107930af08 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:09:05.068440 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84d211b6-2fb3-4a51-a715-e6107930af08 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:09:05.071061 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84d211b6-2fb3-4a51-a715-e6107930af08 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:09:05.071061 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84d211b6-2fb3-4a51-a715-e6107930af08 admin admin] PATH_INFO: `/v3/projects/c57bc7a319f14bf393e6249b2cb59c69` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:09:05.071061 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84d211b6-2fb3-4a51-a715-e6107930af08 admin admin] RBAC: Authorizing `identity:delete_project(project_id=c57bc7a319f14bf393e6249b2cb59c69)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:09:05.072236 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84d211b6-2fb3-4a51-a715-e6107930af08 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:09:05.235471 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=86,keystone:DELETE=13,keystone:INSERT=1 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:09:05.242492 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-84d211b6-2fb3-4a51-a715-e6107930af08 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'c57bc7a319f14bf393e6249b2cb59c69', 'request_id': 'req-84d211b6-2fb3-4a51-a715-e6107930af08'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:09:05.248055 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-84d211b6-2fb3-4a51-a715-e6107930af08 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'c57bc7a319f14bf393e6249b2cb59c69', 'request_id': 'req-84d211b6-2fb3-4a51-a715-e6107930af08'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:09:05.256678 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-84d211b6-2fb3-4a51-a715-e6107930af08 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'c57bc7a319f14bf393e6249b2cb59c69', 'request_id': 'req-84d211b6-2fb3-4a51-a715-e6107930af08'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:09:05.366834 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-84d211b6-2fb3-4a51-a715-e6107930af08 admin admin] The token cache is being invalidate because project c57bc7a319f14bf393e6249b2cb59c69 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81577) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 30 02:09:05.367827 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-84d211b6-2fb3-4a51-a715-e6107930af08 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:09:05.368546 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-84d211b6-2fb3-4a51-a715-e6107930af08 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:09:05.370351 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1260/2516] 10.222.0.27 () {58 vars in 1233 bytes} [Tue Aug 30 02:09:04 2022] DELETE /identity/v3/projects/c57bc7a319f14bf393e6249b2cb59c69 => generated 0 bytes in 382 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 02:09:05.375875 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-567bdb56-479a-4610-9165-c879b63db9f4 None None] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:09:05.378990 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-567bdb56-479a-4610-9165-c879b63db9f4 None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:09:05.378990 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-567bdb56-479a-4610-9165-c879b63db9f4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:09:05.396992 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.auth.core [None req-567bdb56-479a-4610-9165-c879b63db9f4 None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81578) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:09:05.398241 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-567bdb56-479a-4610-9165-c879b63db9f4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:09:05.463197 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1257/2517] 10.222.0.27 () {58 vars in 958 bytes} [Tue Aug 30 02:09:05 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 30 02:09:05.471597 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ecf1473-5301-48bc-bd6c-2211088daa96 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:09:05.472213 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ecf1473-5301-48bc-bd6c-2211088daa96 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:09:05.473220 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ecf1473-5301-48bc-bd6c-2211088daa96 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:09:05.503230 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-7ecf1473-5301-48bc-bd6c-2211088daa96 None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:09:05.504557 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-7ecf1473-5301-48bc-bd6c-2211088daa96 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:09:05.514028 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1261/2518] 10.222.0.27 () {58 vars in 958 bytes} [Tue Aug 30 02:09:05 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 44 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 30 02:09:05.518139 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-567bdb56-479a-4610-9165-c879b63db9f4 None None] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:09:05.528258 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-567bdb56-479a-4610-9165-c879b63db9f4 None None] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:09:05.532892 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32be4d63-147a-4a9f-8a50-d1f0b1c37caf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:09:05.533515 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32be4d63-147a-4a9f-8a50-d1f0b1c37caf admin admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:09:05.533796 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32be4d63-147a-4a9f-8a50-d1f0b1c37caf admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:09:05.534064 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32be4d63-147a-4a9f-8a50-d1f0b1c37caf admin admin] PATH_INFO: `/v3/domains` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:09:05.534708 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32be4d63-147a-4a9f-8a50-d1f0b1c37caf admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81578) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 02:09:05.535008 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32be4d63-147a-4a9f-8a50-d1f0b1c37caf admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:09:05.536777 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32be4d63-147a-4a9f-8a50-d1f0b1c37caf admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:09:05.549212 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1258/2519] 10.222.0.27 () {58 vars in 1153 bytes} [Tue Aug 30 02:09:05 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 30 02:09:05.553800 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ecf1473-5301-48bc-bd6c-2211088daa96 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:09:05.563594 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ecf1473-5301-48bc-bd6c-2211088daa96 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:09:05.568355 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60c8c9d4-cdb3-4746-a125-720923a8ad2f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:09:05.570762 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60c8c9d4-cdb3-4746-a125-720923a8ad2f admin admin] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:09:05.570762 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60c8c9d4-cdb3-4746-a125-720923a8ad2f admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:09:05.570762 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60c8c9d4-cdb3-4746-a125-720923a8ad2f admin admin] PATH_INFO: `/v3/projects` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:09:05.570762 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60c8c9d4-cdb3-4746-a125-720923a8ad2f admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:09:05.572300 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60c8c9d4-cdb3-4746-a125-720923a8ad2f admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:09:05.690000 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1262/2520] 10.222.0.27 () {60 vars in 1153 bytes} [Tue Aug 30 02:09:05 2022] POST /identity/v3/projects => generated 367 bytes in 134 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 02:09:05.691882 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32be4d63-147a-4a9f-8a50-d1f0b1c37caf admin admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:09:05.707006 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32be4d63-147a-4a9f-8a50-d1f0b1c37caf admin admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:09:05.778505 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-331af3e0-1d83-41fc-88fa-cfe0d4873fa9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:09:05.779123 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-331af3e0-1d83-41fc-88fa-cfe0d4873fa9 admin admin] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:09:05.779405 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-331af3e0-1d83-41fc-88fa-cfe0d4873fa9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:09:05.779676 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-331af3e0-1d83-41fc-88fa-cfe0d4873fa9 admin admin] PATH_INFO: `/v3/users` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:09:05.780322 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-331af3e0-1d83-41fc-88fa-cfe0d4873fa9 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:09:05.782265 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-331af3e0-1d83-41fc-88fa-cfe0d4873fa9 admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:09:05.891180 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1259/2521] 10.222.0.27 () {60 vars in 1144 bytes} [Tue Aug 30 02:09:05 2022] POST /identity/v3/users => generated 284 bytes in 196 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 02:09:05.893495 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60c8c9d4-cdb3-4746-a125-720923a8ad2f admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:09:05.903927 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60c8c9d4-cdb3-4746-a125-720923a8ad2f admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:09:05.908454 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa29dbdc-4d23-4ba7-a1f2-e5e943975dde admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:09:05.909065 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa29dbdc-4d23-4ba7-a1f2-e5e943975dde admin admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:09:05.909342 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa29dbdc-4d23-4ba7-a1f2-e5e943975dde admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:09:05.910131 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa29dbdc-4d23-4ba7-a1f2-e5e943975dde admin admin] PATH_INFO: `/v3/roles` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:09:05.910868 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa29dbdc-4d23-4ba7-a1f2-e5e943975dde admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:09:05.912554 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa29dbdc-4d23-4ba7-a1f2-e5e943975dde admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:09:05.924111 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1263/2522] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Aug 30 02:09:05 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 02:09:05.928191 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-331af3e0-1d83-41fc-88fa-cfe0d4873fa9 admin admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:09:05.939028 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-331af3e0-1d83-41fc-88fa-cfe0d4873fa9 admin admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:09:05.943705 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf0b039d-2161-4f92-8878-f07c7177f078 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:09:05.944307 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf0b039d-2161-4f92-8878-f07c7177f078 admin admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:09:05.944585 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf0b039d-2161-4f92-8878-f07c7177f078 admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:09:05.944855 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf0b039d-2161-4f92-8878-f07c7177f078 admin admin] PATH_INFO: `/v3/roles` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:09:05.945531 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf0b039d-2161-4f92-8878-f07c7177f078 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:09:05.947118 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf0b039d-2161-4f92-8878-f07c7177f078 admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:09:05.958253 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1260/2523] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Aug 30 02:09:05 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 02:09:05.967639 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa29dbdc-4d23-4ba7-a1f2-e5e943975dde admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:09:05.978343 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa29dbdc-4d23-4ba7-a1f2-e5e943975dde admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:09:05.984135 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d48d7b8e-50c6-4762-8343-c54d4f80799c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:09:05.984801 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d48d7b8e-50c6-4762-8343-c54d4f80799c admin admin] REQUEST_METHOD: `PUT` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:09:05.985082 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d48d7b8e-50c6-4762-8343-c54d4f80799c admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:09:05.985354 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d48d7b8e-50c6-4762-8343-c54d4f80799c admin admin] PATH_INFO: `/v3/projects/08eae140daa04d3f8ea47e9bfc6a21ff/users/c5ea09c8442e4186b10c30a556e2f7ea/roles/1fb81afcb9f245f3b989add94598a063` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:09:06.001780 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d48d7b8e-50c6-4762-8343-c54d4f80799c admin admin] RBAC: Authorizing `identity:create_grant(project_id=08eae140daa04d3f8ea47e9bfc6a21ff, user_id=c5ea09c8442e4186b10c30a556e2f7ea, role_id=1fb81afcb9f245f3b989add94598a063)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:09:06.004436 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d48d7b8e-50c6-4762-8343-c54d4f80799c admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:09:06.115969 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1264/2524] 10.222.0.27 () {60 vars in 1483 bytes} [Tue Aug 30 02:09:05 2022] PUT /identity/v3/projects/08eae140daa04d3f8ea47e9bfc6a21ff/users/c5ea09c8442e4186b10c30a556e2f7ea/roles/1fb81afcb9f245f3b989add94598a063 => generated 0 bytes in 149 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 02:09:06.126359 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31cda906-8cab-43dc-8706-e8d128ee6f50 None None] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:09:06.126646 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31cda906-8cab-43dc-8706-e8d128ee6f50 None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:09:06.126919 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31cda906-8cab-43dc-8706-e8d128ee6f50 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:09:06.152692 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:09:06.157680 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.auth.core [None req-31cda906-8cab-43dc-8706-e8d128ee6f50 None None] MFA Rules not processed for user `c5ea09c8442e4186b10c30a556e2f7ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81578) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:09:06.158901 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-31cda906-8cab-43dc-8706-e8d128ee6f50 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:09:06.225938 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1261/2525] 10.222.0.27 () {58 vars in 958 bytes} [Tue Aug 30 02:09:06 2022] POST /identity/v3/auth/tokens => generated 3244 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 02:09:06.238296 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d48d7b8e-50c6-4762-8343-c54d4f80799c admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:09:06.239959 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-d48d7b8e-50c6-4762-8343-c54d4f80799c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:09:06.298407 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d48d7b8e-50c6-4762-8343-c54d4f80799c admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:09:06.302146 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcb2ee20-9d5c-4958-a292-feb6820d338d service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ab936b3767a8417aa80c8c8f1b84b667', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4562f3951fbe465280073e60a11e44e2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:09:06.302758 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcb2ee20-9d5c-4958-a292-feb6820d338d service cinder] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:09:06.303042 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcb2ee20-9d5c-4958-a292-feb6820d338d service cinder] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:09:06.303319 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcb2ee20-9d5c-4958-a292-feb6820d338d service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:09:06.311687 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fcb2ee20-9d5c-4958-a292-feb6820d338d service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:09:06.313133 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fcb2ee20-9d5c-4958-a292-feb6820d338d service cinder] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:09:06.321770 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1265/2526] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Aug 30 02:09:06 2022] GET /identity/v3/auth/tokens => generated 3244 bytes in 84 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 02:09:09.606024 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31cda906-8cab-43dc-8706-e8d128ee6f50 None None] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:09:09.607460 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-31cda906-8cab-43dc-8706-e8d128ee6f50 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:09:09.795213 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31cda906-8cab-43dc-8706-e8d128ee6f50 None None] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:09:09.798915 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb67a9ca-6658-4b54-a64a-e411688a6c55 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f3ea1580c56f4814a92bbb2acf542905', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4562f3951fbe465280073e60a11e44e2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:09:09.799521 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb67a9ca-6658-4b54-a64a-e411688a6c55 service neutron] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:09:09.799806 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb67a9ca-6658-4b54-a64a-e411688a6c55 service neutron] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:09:09.800074 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb67a9ca-6658-4b54-a64a-e411688a6c55 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:09:09.801825 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-cb67a9ca-6658-4b54-a64a-e411688a6c55 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:09:09.864706 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb67a9ca-6658-4b54-a64a-e411688a6c55 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:09:09.866194 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb67a9ca-6658-4b54-a64a-e411688a6c55 service neutron] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:09:09.875622 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1262/2527] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Aug 30 02:09:09 2022] GET /identity/v3/auth/tokens => generated 3315 bytes in 271 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 02:09:09.942298 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcb2ee20-9d5c-4958-a292-feb6820d338d service cinder] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:09:09.951817 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcb2ee20-9d5c-4958-a292-feb6820d338d service cinder] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:09:09.955732 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a46c9fed-f7d4-42f9-9b72-2e64bbff8aea service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ab936b3767a8417aa80c8c8f1b84b667', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4562f3951fbe465280073e60a11e44e2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:09:09.956333 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a46c9fed-f7d4-42f9-9b72-2e64bbff8aea service cinder] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:09:09.956616 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a46c9fed-f7d4-42f9-9b72-2e64bbff8aea service cinder] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:09:09.956890 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a46c9fed-f7d4-42f9-9b72-2e64bbff8aea service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:09:09.958675 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-a46c9fed-f7d4-42f9-9b72-2e64bbff8aea service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:09:10.023851 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a46c9fed-f7d4-42f9-9b72-2e64bbff8aea service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:09:10.025287 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a46c9fed-f7d4-42f9-9b72-2e64bbff8aea service cinder] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:09:10.035044 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1266/2528] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Aug 30 02:09:09 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 93 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 02:09:14.382266 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb67a9ca-6658-4b54-a64a-e411688a6c55 service neutron] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:09:14.389150 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-cb67a9ca-6658-4b54-a64a-e411688a6c55 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:09:14.460822 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:09:14.474010 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb67a9ca-6658-4b54-a64a-e411688a6c55 service neutron] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:09:14.479145 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-797f80a3-691b-413d-a76b-566c19205bf2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:09:14.479863 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-797f80a3-691b-413d-a76b-566c19205bf2 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:09:14.480208 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-797f80a3-691b-413d-a76b-566c19205bf2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:09:14.480528 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-797f80a3-691b-413d-a76b-566c19205bf2 admin admin] PATH_INFO: `/v3/users/39e161dfd48b49d783fa39ec2dc599d7` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:09:14.481640 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-797f80a3-691b-413d-a76b-566c19205bf2 admin admin] RBAC: Authorizing `identity:delete_user(user_id=39e161dfd48b49d783fa39ec2dc599d7)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:09:14.483651 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-797f80a3-691b-413d-a76b-566c19205bf2 admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:09:14.666899 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-797f80a3-691b-413d-a76b-566c19205bf2 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '39e161dfd48b49d783fa39ec2dc599d7'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:09:14.669013 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-797f80a3-691b-413d-a76b-566c19205bf2 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '39e161dfd48b49d783fa39ec2dc599d7'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:09:14.749929 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-797f80a3-691b-413d-a76b-566c19205bf2 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '39e161dfd48b49d783fa39ec2dc599d7'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:09:14.750633 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-797f80a3-691b-413d-a76b-566c19205bf2 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '39e161dfd48b49d783fa39ec2dc599d7'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:09:14.759039 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-797f80a3-691b-413d-a76b-566c19205bf2 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '39e161dfd48b49d783fa39ec2dc599d7'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:09:14.778275 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1263/2529] 10.222.0.27 () {58 vars in 1224 bytes} [Tue Aug 30 02:09:14 2022] DELETE /identity/v3/users/39e161dfd48b49d783fa39ec2dc599d7 => generated 0 bytes in 397 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 02:09:14.783052 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f7fb9a4-0627-408f-ae25-5bde68f5469b None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:09:14.783535 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f7fb9a4-0627-408f-ae25-5bde68f5469b None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:09:14.783631 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f7fb9a4-0627-408f-ae25-5bde68f5469b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:09:14.791645 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=93,keystone:DELETE=4,keystone:INSERT=3 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:09:14.803049 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-1f7fb9a4-0627-408f-ae25-5bde68f5469b None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:09:14.804271 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-1f7fb9a4-0627-408f-ae25-5bde68f5469b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:09:14.864974 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1267/2530] 10.222.0.27 () {58 vars in 958 bytes} [Tue Aug 30 02:09:14 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 30 02:09:14.875601 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-797f80a3-691b-413d-a76b-566c19205bf2 admin admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:09:14.877042 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-797f80a3-691b-413d-a76b-566c19205bf2 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:09:14.940999 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-797f80a3-691b-413d-a76b-566c19205bf2 admin admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:09:14.944581 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64525c07-c75d-4dc6-9299-fdfa7822aa52 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f3ea1580c56f4814a92bbb2acf542905', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4562f3951fbe465280073e60a11e44e2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:09:14.945189 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64525c07-c75d-4dc6-9299-fdfa7822aa52 service neutron] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:09:14.945476 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64525c07-c75d-4dc6-9299-fdfa7822aa52 service neutron] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:09:14.945765 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64525c07-c75d-4dc6-9299-fdfa7822aa52 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:09:14.954957 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64525c07-c75d-4dc6-9299-fdfa7822aa52 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:09:14.956358 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64525c07-c75d-4dc6-9299-fdfa7822aa52 service neutron] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:09:14.966013 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1264/2531] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Aug 30 02:09:14 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 91 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 02:09:15.403628 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f7fb9a4-0627-408f-ae25-5bde68f5469b None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:09:15.405194 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-1f7fb9a4-0627-408f-ae25-5bde68f5469b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:09:15.417125 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f7fb9a4-0627-408f-ae25-5bde68f5469b None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:09:15.423053 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99b47303-b731-4819-9703-67f8cbfc0457 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:09:15.423694 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99b47303-b731-4819-9703-67f8cbfc0457 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:09:15.430760 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99b47303-b731-4819-9703-67f8cbfc0457 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:09:15.430760 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99b47303-b731-4819-9703-67f8cbfc0457 admin admin] PATH_INFO: `/v3/projects/3e1069cdcd5b4c4c93a792ba86928e79` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:09:15.430760 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99b47303-b731-4819-9703-67f8cbfc0457 admin admin] RBAC: Authorizing `identity:delete_project(project_id=3e1069cdcd5b4c4c93a792ba86928e79)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:09:15.430760 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99b47303-b731-4819-9703-67f8cbfc0457 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:09:15.548690 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-99b47303-b731-4819-9703-67f8cbfc0457 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '3e1069cdcd5b4c4c93a792ba86928e79', 'request_id': 'req-99b47303-b731-4819-9703-67f8cbfc0457'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:09:15.554475 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-99b47303-b731-4819-9703-67f8cbfc0457 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '3e1069cdcd5b4c4c93a792ba86928e79', 'request_id': 'req-99b47303-b731-4819-9703-67f8cbfc0457'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:09:15.561146 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-99b47303-b731-4819-9703-67f8cbfc0457 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '3e1069cdcd5b4c4c93a792ba86928e79', 'request_id': 'req-99b47303-b731-4819-9703-67f8cbfc0457'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:09:15.620798 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-99b47303-b731-4819-9703-67f8cbfc0457 admin admin] The token cache is being invalidate because project 3e1069cdcd5b4c4c93a792ba86928e79 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81577) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 30 02:09:15.621261 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-99b47303-b731-4819-9703-67f8cbfc0457 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:09:15.621912 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-99b47303-b731-4819-9703-67f8cbfc0457 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:09:15.623399 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1268/2532] 10.222.0.27 () {58 vars in 1233 bytes} [Tue Aug 30 02:09:15 2022] DELETE /identity/v3/projects/3e1069cdcd5b4c4c93a792ba86928e79 => generated 0 bytes in 220 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 02:09:15.627429 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64525c07-c75d-4dc6-9299-fdfa7822aa52 service neutron] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:09:15.637808 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64525c07-c75d-4dc6-9299-fdfa7822aa52 service neutron] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:09:15.697169 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1d070f0-663e-4b49-b6af-6c20354a3132 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:09:15.697859 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1d070f0-663e-4b49-b6af-6c20354a3132 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:09:15.698157 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1d070f0-663e-4b49-b6af-6c20354a3132 admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:09:15.698942 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1d070f0-663e-4b49-b6af-6c20354a3132 admin admin] PATH_INFO: `/v3/users/c9f80f8047a14665afd2de282bad7ed3` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:09:15.700293 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1d070f0-663e-4b49-b6af-6c20354a3132 admin admin] RBAC: Authorizing `identity:delete_user(user_id=c9f80f8047a14665afd2de282bad7ed3)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:09:15.702239 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1d070f0-663e-4b49-b6af-6c20354a3132 admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:09:15.721030 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=88,keystone:DELETE=11,keystone:INSERT=1 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:09:15.986739 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-a1d070f0-663e-4b49-b6af-6c20354a3132 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'c9f80f8047a14665afd2de282bad7ed3'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:09:15.987391 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-a1d070f0-663e-4b49-b6af-6c20354a3132 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'c9f80f8047a14665afd2de282bad7ed3'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:09:16.035772 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-a1d070f0-663e-4b49-b6af-6c20354a3132 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'c9f80f8047a14665afd2de282bad7ed3'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:09:16.036393 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-a1d070f0-663e-4b49-b6af-6c20354a3132 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'c9f80f8047a14665afd2de282bad7ed3'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:09:16.043693 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-a1d070f0-663e-4b49-b6af-6c20354a3132 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'c9f80f8047a14665afd2de282bad7ed3'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:09:16.062168 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1265/2533] 10.222.0.27 () {58 vars in 1224 bytes} [Tue Aug 30 02:09:15 2022] DELETE /identity/v3/users/c9f80f8047a14665afd2de282bad7ed3 => generated 0 bytes in 435 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 02:09:16.482547 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99b47303-b731-4819-9703-67f8cbfc0457 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:09:16.484114 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-99b47303-b731-4819-9703-67f8cbfc0457 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:09:16.502171 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99b47303-b731-4819-9703-67f8cbfc0457 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:09:16.562861 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba1546cb-4089-461b-934f-1a707f45b53b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:09:16.563504 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba1546cb-4089-461b-934f-1a707f45b53b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:09:16.564751 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba1546cb-4089-461b-934f-1a707f45b53b admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:09:16.564751 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba1546cb-4089-461b-934f-1a707f45b53b admin admin] PATH_INFO: `/v3/projects/5df0340c865e422c8c9b8b6aeaa4a9a7` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:09:16.565826 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba1546cb-4089-461b-934f-1a707f45b53b admin admin] RBAC: Authorizing `identity:delete_project(project_id=5df0340c865e422c8c9b8b6aeaa4a9a7)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:09:16.567766 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba1546cb-4089-461b-934f-1a707f45b53b admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:09:16.585922 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1d070f0-663e-4b49-b6af-6c20354a3132 admin admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:09:16.587294 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-a1d070f0-663e-4b49-b6af-6c20354a3132 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:09:16.600676 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1d070f0-663e-4b49-b6af-6c20354a3132 admin admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:09:16.605726 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-593e1dfd-eb49-44fe-b0e9-00e3bb8d1d34 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:09:16.606549 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-593e1dfd-eb49-44fe-b0e9-00e3bb8d1d34 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:09:16.606838 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-593e1dfd-eb49-44fe-b0e9-00e3bb8d1d34 admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:09:16.607098 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-593e1dfd-eb49-44fe-b0e9-00e3bb8d1d34 admin admin] PATH_INFO: `/v3/users/c5ea09c8442e4186b10c30a556e2f7ea` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:09:16.608199 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-593e1dfd-eb49-44fe-b0e9-00e3bb8d1d34 admin admin] RBAC: Authorizing `identity:delete_user(user_id=c5ea09c8442e4186b10c30a556e2f7ea)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:09:16.610315 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-593e1dfd-eb49-44fe-b0e9-00e3bb8d1d34 admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:09:16.701866 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=90,keystone:DELETE=9,keystone:INSERT=1 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:09:16.714206 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-ba1546cb-4089-461b-934f-1a707f45b53b admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '5df0340c865e422c8c9b8b6aeaa4a9a7', 'request_id': 'req-ba1546cb-4089-461b-934f-1a707f45b53b'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:09:16.720798 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-ba1546cb-4089-461b-934f-1a707f45b53b admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '5df0340c865e422c8c9b8b6aeaa4a9a7', 'request_id': 'req-ba1546cb-4089-461b-934f-1a707f45b53b'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:09:16.727613 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-ba1546cb-4089-461b-934f-1a707f45b53b admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '5df0340c865e422c8c9b8b6aeaa4a9a7', 'request_id': 'req-ba1546cb-4089-461b-934f-1a707f45b53b'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:09:16.815238 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-ba1546cb-4089-461b-934f-1a707f45b53b admin admin] The token cache is being invalidate because project 5df0340c865e422c8c9b8b6aeaa4a9a7 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81577) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 30 02:09:16.815622 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-ba1546cb-4089-461b-934f-1a707f45b53b admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:09:16.816279 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-ba1546cb-4089-461b-934f-1a707f45b53b admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:09:16.818101 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1269/2534] 10.222.0.27 () {58 vars in 1233 bytes} [Tue Aug 30 02:09:16 2022] DELETE /identity/v3/projects/5df0340c865e422c8c9b8b6aeaa4a9a7 => generated 0 bytes in 336 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 02:09:16.821587 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba1546cb-4089-461b-934f-1a707f45b53b admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:09:16.822902 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-ba1546cb-4089-461b-934f-1a707f45b53b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:09:16.825281 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-593e1dfd-eb49-44fe-b0e9-00e3bb8d1d34 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'c5ea09c8442e4186b10c30a556e2f7ea'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:09:16.828823 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-593e1dfd-eb49-44fe-b0e9-00e3bb8d1d34 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'c5ea09c8442e4186b10c30a556e2f7ea'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:09:16.889715 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba1546cb-4089-461b-934f-1a707f45b53b admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:09:16.894591 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68c78a09-51b0-4973-a3c8-2bdac1849a8b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:09:16.895225 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68c78a09-51b0-4973-a3c8-2bdac1849a8b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:09:16.895502 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68c78a09-51b0-4973-a3c8-2bdac1849a8b admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:09:16.895775 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68c78a09-51b0-4973-a3c8-2bdac1849a8b admin admin] PATH_INFO: `/v3/users/0da22da0af7449aeb838f9219c8add65` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:09:16.896984 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68c78a09-51b0-4973-a3c8-2bdac1849a8b admin admin] RBAC: Authorizing `identity:delete_user(user_id=0da22da0af7449aeb838f9219c8add65)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:09:16.898930 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68c78a09-51b0-4973-a3c8-2bdac1849a8b admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:09:16.907606 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-593e1dfd-eb49-44fe-b0e9-00e3bb8d1d34 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'c5ea09c8442e4186b10c30a556e2f7ea'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:09:16.908282 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-593e1dfd-eb49-44fe-b0e9-00e3bb8d1d34 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'c5ea09c8442e4186b10c30a556e2f7ea'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:09:16.917576 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-593e1dfd-eb49-44fe-b0e9-00e3bb8d1d34 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'c5ea09c8442e4186b10c30a556e2f7ea'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:09:16.936840 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1266/2535] 10.222.0.27 () {58 vars in 1224 bytes} [Tue Aug 30 02:09:16 2022] DELETE /identity/v3/users/c5ea09c8442e4186b10c30a556e2f7ea => generated 0 bytes in 351 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 30 02:09:16.941073 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5723a23a-5135-4ea2-be1b-5967d69c6097 None None] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:09:16.941338 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5723a23a-5135-4ea2-be1b-5967d69c6097 None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:09:16.941675 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5723a23a-5135-4ea2-be1b-5967d69c6097 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:09:16.961600 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.auth.core [None req-5723a23a-5135-4ea2-be1b-5967d69c6097 None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81578) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:09:16.963509 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-5723a23a-5135-4ea2-be1b-5967d69c6097 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:09:17.025820 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1267/2536] 10.222.0.27 () {58 vars in 958 bytes} [Tue Aug 30 02:09:16 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 30 02:09:17.036061 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5723a23a-5135-4ea2-be1b-5967d69c6097 None None] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:09:17.038264 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-5723a23a-5135-4ea2-be1b-5967d69c6097 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:09:17.047343 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:DELETE=19,keystone:SELECT=79,keystone:INSERT=2 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:09:17.075892 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-68c78a09-51b0-4973-a3c8-2bdac1849a8b admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '0da22da0af7449aeb838f9219c8add65'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:09:17.076664 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-68c78a09-51b0-4973-a3c8-2bdac1849a8b admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '0da22da0af7449aeb838f9219c8add65'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:09:17.077522 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-68c78a09-51b0-4973-a3c8-2bdac1849a8b admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '0da22da0af7449aeb838f9219c8add65'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:09:17.092302 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-68c78a09-51b0-4973-a3c8-2bdac1849a8b admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '0da22da0af7449aeb838f9219c8add65'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:09:17.100379 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-68c78a09-51b0-4973-a3c8-2bdac1849a8b admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '0da22da0af7449aeb838f9219c8add65'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:09:17.103433 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5723a23a-5135-4ea2-be1b-5967d69c6097 None None] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:09:17.107779 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5110d17e-288e-482d-b40e-ef3e90eb4bca service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f3ea1580c56f4814a92bbb2acf542905', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4562f3951fbe465280073e60a11e44e2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:09:17.108590 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5110d17e-288e-482d-b40e-ef3e90eb4bca service neutron] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:09:17.108936 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5110d17e-288e-482d-b40e-ef3e90eb4bca service neutron] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:09:17.109303 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5110d17e-288e-482d-b40e-ef3e90eb4bca service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:09:17.111398 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-5110d17e-288e-482d-b40e-ef3e90eb4bca service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:09:17.125160 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5110d17e-288e-482d-b40e-ef3e90eb4bca service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:09:17.126612 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5110d17e-288e-482d-b40e-ef3e90eb4bca service neutron] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:09:17.136102 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1268/2537] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Aug 30 02:09:17 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 101 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 30 02:09:17.218256 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1270/2538] 10.222.0.27 () {58 vars in 1224 bytes} [Tue Aug 30 02:09:16 2022] DELETE /identity/v3/users/0da22da0af7449aeb838f9219c8add65 => generated 0 bytes in 397 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 30 02:09:17.607012 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5110d17e-288e-482d-b40e-ef3e90eb4bca service neutron] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:09:17.608434 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-5110d17e-288e-482d-b40e-ef3e90eb4bca service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:09:17.620457 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5110d17e-288e-482d-b40e-ef3e90eb4bca service neutron] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:09:17.669534 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68c78a09-51b0-4973-a3c8-2bdac1849a8b admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:09:17.671850 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-68c78a09-51b0-4973-a3c8-2bdac1849a8b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:09:17.680492 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa0c5e31-2f6a-4e3c-be27-48ef2ad09a1f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:09:17.681142 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa0c5e31-2f6a-4e3c-be27-48ef2ad09a1f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:09:17.681595 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa0c5e31-2f6a-4e3c-be27-48ef2ad09a1f admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:09:17.681712 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa0c5e31-2f6a-4e3c-be27-48ef2ad09a1f admin admin] PATH_INFO: `/v3/projects/08eae140daa04d3f8ea47e9bfc6a21ff` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:09:17.682847 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa0c5e31-2f6a-4e3c-be27-48ef2ad09a1f admin admin] RBAC: Authorizing `identity:delete_project(project_id=08eae140daa04d3f8ea47e9bfc6a21ff)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:09:17.685376 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa0c5e31-2f6a-4e3c-be27-48ef2ad09a1f admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:09:17.726668 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=83,keystone:DELETE=15,keystone:INSERT=2 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:09:17.749466 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68c78a09-51b0-4973-a3c8-2bdac1849a8b admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:09:17.754248 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb13d1f3-4cf2-49f8-97af-e85fe932e63c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:09:17.754904 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb13d1f3-4cf2-49f8-97af-e85fe932e63c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:09:17.755186 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb13d1f3-4cf2-49f8-97af-e85fe932e63c admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:09:17.755459 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb13d1f3-4cf2-49f8-97af-e85fe932e63c admin admin] PATH_INFO: `/v3/projects/58b53773ef1b45c3a467233747c4f75e` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:09:17.756571 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb13d1f3-4cf2-49f8-97af-e85fe932e63c admin admin] RBAC: Authorizing `identity:delete_project(project_id=58b53773ef1b45c3a467233747c4f75e)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:09:17.758539 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb13d1f3-4cf2-49f8-97af-e85fe932e63c admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:09:17.819030 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-fa0c5e31-2f6a-4e3c-be27-48ef2ad09a1f admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '08eae140daa04d3f8ea47e9bfc6a21ff', 'request_id': 'req-fa0c5e31-2f6a-4e3c-be27-48ef2ad09a1f'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:09:17.824793 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-fa0c5e31-2f6a-4e3c-be27-48ef2ad09a1f admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '08eae140daa04d3f8ea47e9bfc6a21ff', 'request_id': 'req-fa0c5e31-2f6a-4e3c-be27-48ef2ad09a1f'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:09:17.832718 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-fa0c5e31-2f6a-4e3c-be27-48ef2ad09a1f admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '08eae140daa04d3f8ea47e9bfc6a21ff', 'request_id': 'req-fa0c5e31-2f6a-4e3c-be27-48ef2ad09a1f'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:09:17.910501 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-bb13d1f3-4cf2-49f8-97af-e85fe932e63c admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '58b53773ef1b45c3a467233747c4f75e', 'request_id': 'req-bb13d1f3-4cf2-49f8-97af-e85fe932e63c'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:09:17.916054 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-bb13d1f3-4cf2-49f8-97af-e85fe932e63c admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '58b53773ef1b45c3a467233747c4f75e', 'request_id': 'req-bb13d1f3-4cf2-49f8-97af-e85fe932e63c'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:09:17.923286 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-bb13d1f3-4cf2-49f8-97af-e85fe932e63c admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '58b53773ef1b45c3a467233747c4f75e', 'request_id': 'req-bb13d1f3-4cf2-49f8-97af-e85fe932e63c'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:09:17.926222 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-fa0c5e31-2f6a-4e3c-be27-48ef2ad09a1f admin admin] The token cache is being invalidate because project 08eae140daa04d3f8ea47e9bfc6a21ff was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81578) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 30 02:09:17.932388 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-fa0c5e31-2f6a-4e3c-be27-48ef2ad09a1f admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:09:17.932388 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-fa0c5e31-2f6a-4e3c-be27-48ef2ad09a1f admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:09:17.932388 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1269/2539] 10.222.0.27 () {58 vars in 1233 bytes} [Tue Aug 30 02:09:17 2022] DELETE /identity/v3/projects/08eae140daa04d3f8ea47e9bfc6a21ff => generated 0 bytes in 322 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 02:09:17.933976 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fdc5f96-ae23-4fa5-be78-169a45b41ecc None None] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:09:17.934315 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fdc5f96-ae23-4fa5-be78-169a45b41ecc None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:09:17.934759 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fdc5f96-ae23-4fa5-be78-169a45b41ecc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:09:17.956527 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.auth.core [None req-6fdc5f96-ae23-4fa5-be78-169a45b41ecc None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81578) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:09:17.961095 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-6fdc5f96-ae23-4fa5-be78-169a45b41ecc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:09:17.990479 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:09:18.016008 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1270/2540] 10.222.0.27 () {58 vars in 958 bytes} [Tue Aug 30 02:09:17 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 30 02:09:18.025280 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9b7a873-1392-43f5-bbd1-81759661b581 None None] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:09:18.025610 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9b7a873-1392-43f5-bbd1-81759661b581 None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:09:18.025881 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9b7a873-1392-43f5-bbd1-81759661b581 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:09:18.029335 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-bb13d1f3-4cf2-49f8-97af-e85fe932e63c admin admin] The token cache is being invalidate because project 58b53773ef1b45c3a467233747c4f75e was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81577) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 30 02:09:18.029664 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-bb13d1f3-4cf2-49f8-97af-e85fe932e63c admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:09:18.030297 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-bb13d1f3-4cf2-49f8-97af-e85fe932e63c admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:09:18.032204 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1271/2541] 10.222.0.27 () {58 vars in 1233 bytes} [Tue Aug 30 02:09:17 2022] DELETE /identity/v3/projects/58b53773ef1b45c3a467233747c4f75e => generated 0 bytes in 363 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 02:09:18.038588 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e31b6e7e-6fbb-414b-8eca-a022d016ed3c None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:09:18.038893 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e31b6e7e-6fbb-414b-8eca-a022d016ed3c None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:09:18.039172 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e31b6e7e-6fbb-414b-8eca-a022d016ed3c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:09:18.046759 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.auth.core [None req-a9b7a873-1392-43f5-bbd1-81759661b581 None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81578) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:09:18.047957 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-a9b7a873-1392-43f5-bbd1-81759661b581 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:09:18.058031 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1271/2542] 10.222.0.27 () {58 vars in 958 bytes} [Tue Aug 30 02:09:18 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 30 02:09:18.062105 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-e31b6e7e-6fbb-414b-8eca-a022d016ed3c None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:09:18.062863 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9b7a873-1392-43f5-bbd1-81759661b581 None None] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:09:18.063335 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-e31b6e7e-6fbb-414b-8eca-a022d016ed3c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:09:18.071418 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1272/2543] 10.222.0.27 () {58 vars in 958 bytes} [Tue Aug 30 02:09:18 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 30 02:09:18.071741 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9b7a873-1392-43f5-bbd1-81759661b581 None None] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:09:18.076404 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8376787-c673-4c8e-8db1-a41e73d05c8c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:09:18.076996 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8376787-c673-4c8e-8db1-a41e73d05c8c admin admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:09:18.077269 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8376787-c673-4c8e-8db1-a41e73d05c8c admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:09:18.077651 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8376787-c673-4c8e-8db1-a41e73d05c8c admin admin] PATH_INFO: `/v3/domains` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:09:18.078297 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8376787-c673-4c8e-8db1-a41e73d05c8c admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81578) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 02:09:18.078593 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8376787-c673-4c8e-8db1-a41e73d05c8c admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:09:18.080426 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8376787-c673-4c8e-8db1-a41e73d05c8c admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:09:18.081224 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd2cc08b-9385-4164-8ceb-1a769b5d6a74 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:09:18.081532 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd2cc08b-9385-4164-8ceb-1a769b5d6a74 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:09:18.081838 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd2cc08b-9385-4164-8ceb-1a769b5d6a74 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:09:18.095944 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1272/2544] 10.222.0.27 () {58 vars in 1153 bytes} [Tue Aug 30 02:09:18 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 30 02:09:18.097208 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8376787-c673-4c8e-8db1-a41e73d05c8c admin admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:09:18.101888 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-bd2cc08b-9385-4164-8ceb-1a769b5d6a74 None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:09:18.103351 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-bd2cc08b-9385-4164-8ceb-1a769b5d6a74 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:09:18.108287 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8376787-c673-4c8e-8db1-a41e73d05c8c admin admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:09:18.115409 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1273/2545] 10.222.0.27 () {58 vars in 958 bytes} [Tue Aug 30 02:09:18 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 30 02:09:18.116752 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e589779c-1d86-4de5-94c8-5c341ba8b7d3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:09:18.116752 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e589779c-1d86-4de5-94c8-5c341ba8b7d3 admin admin] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:09:18.116752 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e589779c-1d86-4de5-94c8-5c341ba8b7d3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:09:18.116752 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e589779c-1d86-4de5-94c8-5c341ba8b7d3 admin admin] PATH_INFO: `/v3/projects` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:09:18.117150 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e589779c-1d86-4de5-94c8-5c341ba8b7d3 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:09:18.117242 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd2cc08b-9385-4164-8ceb-1a769b5d6a74 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:09:18.117332 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e589779c-1d86-4de5-94c8-5c341ba8b7d3 admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:09:18.126242 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd2cc08b-9385-4164-8ceb-1a769b5d6a74 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:09:18.131229 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65910f91-b1b7-4f20-8bee-e97eab067699 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:09:18.131832 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65910f91-b1b7-4f20-8bee-e97eab067699 admin admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:09:18.132113 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65910f91-b1b7-4f20-8bee-e97eab067699 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:09:18.132377 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65910f91-b1b7-4f20-8bee-e97eab067699 admin admin] PATH_INFO: `/v3/domains` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:09:18.133116 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65910f91-b1b7-4f20-8bee-e97eab067699 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 02:09:18.133417 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65910f91-b1b7-4f20-8bee-e97eab067699 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:09:18.135182 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65910f91-b1b7-4f20-8bee-e97eab067699 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:09:18.150057 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1274/2546] 10.222.0.27 () {58 vars in 1153 bytes} [Tue Aug 30 02:09:18 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 02:09:18.154734 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65910f91-b1b7-4f20-8bee-e97eab067699 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:09:18.164848 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65910f91-b1b7-4f20-8bee-e97eab067699 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:09:18.170366 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7dde9100-458e-4c8b-98f1-9f606b611587 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:09:18.170968 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dde9100-458e-4c8b-98f1-9f606b611587 admin admin] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:09:18.171269 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dde9100-458e-4c8b-98f1-9f606b611587 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:09:18.171519 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dde9100-458e-4c8b-98f1-9f606b611587 admin admin] PATH_INFO: `/v3/projects` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:09:18.172169 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7dde9100-458e-4c8b-98f1-9f606b611587 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:09:18.174105 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7dde9100-458e-4c8b-98f1-9f606b611587 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:09:18.197025 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1273/2547] 10.222.0.27 () {60 vars in 1153 bytes} [Tue Aug 30 02:09:18 2022] POST /identity/v3/projects => generated 369 bytes in 100 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 30 02:09:18.201899 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e589779c-1d86-4de5-94c8-5c341ba8b7d3 admin admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:09:18.267152 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e589779c-1d86-4de5-94c8-5c341ba8b7d3 admin admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:09:18.272464 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b35fd81e-c932-44ee-ba91-3c5d8fd85901 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:09:18.273074 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b35fd81e-c932-44ee-ba91-3c5d8fd85901 admin admin] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:09:18.273360 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b35fd81e-c932-44ee-ba91-3c5d8fd85901 admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:09:18.273668 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b35fd81e-c932-44ee-ba91-3c5d8fd85901 admin admin] PATH_INFO: `/v3/users` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:09:18.274420 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b35fd81e-c932-44ee-ba91-3c5d8fd85901 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:09:18.276400 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b35fd81e-c932-44ee-ba91-3c5d8fd85901 admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:09:18.282968 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1275/2548] 10.222.0.27 () {60 vars in 1153 bytes} [Tue Aug 30 02:09:18 2022] POST /identity/v3/projects => generated 377 bytes in 129 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 30 02:09:18.287213 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7dde9100-458e-4c8b-98f1-9f606b611587 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:09:18.352824 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7dde9100-458e-4c8b-98f1-9f606b611587 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:09:18.358393 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6eeacbd0-665d-4ea0-b13c-b1a4766be771 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:09:18.358997 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6eeacbd0-665d-4ea0-b13c-b1a4766be771 admin admin] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:09:18.359278 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6eeacbd0-665d-4ea0-b13c-b1a4766be771 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:09:18.359550 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6eeacbd0-665d-4ea0-b13c-b1a4766be771 admin admin] PATH_INFO: `/v3/users` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:09:18.360172 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6eeacbd0-665d-4ea0-b13c-b1a4766be771 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:09:18.362130 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6eeacbd0-665d-4ea0-b13c-b1a4766be771 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:09:18.370102 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1274/2549] 10.222.0.27 () {60 vars in 1144 bytes} [Tue Aug 30 02:09:18 2022] POST /identity/v3/users => generated 285 bytes in 169 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 30 02:09:18.375071 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b35fd81e-c932-44ee-ba91-3c5d8fd85901 admin admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:09:18.384620 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b35fd81e-c932-44ee-ba91-3c5d8fd85901 admin admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:09:18.389372 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a97fcb7d-c5d0-491a-8208-e3a433d28c5b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:09:18.389985 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a97fcb7d-c5d0-491a-8208-e3a433d28c5b admin admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:09:18.390268 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a97fcb7d-c5d0-491a-8208-e3a433d28c5b admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:09:18.390542 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a97fcb7d-c5d0-491a-8208-e3a433d28c5b admin admin] PATH_INFO: `/v3/roles` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:09:18.391176 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a97fcb7d-c5d0-491a-8208-e3a433d28c5b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:09:18.392744 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a97fcb7d-c5d0-491a-8208-e3a433d28c5b admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:09:18.403217 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1275/2550] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Aug 30 02:09:18 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 30 02:09:18.407049 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a97fcb7d-c5d0-491a-8208-e3a433d28c5b admin admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:09:18.416276 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a97fcb7d-c5d0-491a-8208-e3a433d28c5b admin admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:09:18.421487 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-602ff974-100f-41cd-8e50-ca19669c6a27 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:09:18.422056 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-602ff974-100f-41cd-8e50-ca19669c6a27 admin admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:09:18.422337 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-602ff974-100f-41cd-8e50-ca19669c6a27 admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:09:18.422611 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-602ff974-100f-41cd-8e50-ca19669c6a27 admin admin] PATH_INFO: `/v3/roles` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:09:18.423229 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-602ff974-100f-41cd-8e50-ca19669c6a27 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:09:18.424751 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-602ff974-100f-41cd-8e50-ca19669c6a27 admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:09:18.435446 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1276/2551] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Aug 30 02:09:18 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 30 02:09:18.439362 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-602ff974-100f-41cd-8e50-ca19669c6a27 admin admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:09:18.448545 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-602ff974-100f-41cd-8e50-ca19669c6a27 admin admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:09:18.453286 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6b43bf2-3a8a-4b7d-b30e-4ed585d7cc7e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:09:18.453928 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6b43bf2-3a8a-4b7d-b30e-4ed585d7cc7e admin admin] REQUEST_METHOD: `PUT` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:09:18.454205 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6b43bf2-3a8a-4b7d-b30e-4ed585d7cc7e admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:09:18.454475 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6b43bf2-3a8a-4b7d-b30e-4ed585d7cc7e admin admin] PATH_INFO: `/v3/projects/7b8a54483ae04230ab5e0727c7eae0bc/users/c3557e6b6b2d46909db982ce76bbfdb8/roles/1fb81afcb9f245f3b989add94598a063` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:09:18.461656 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1276/2552] 10.222.0.27 () {60 vars in 1144 bytes} [Tue Aug 30 02:09:18 2022] POST /identity/v3/users => generated 289 bytes in 172 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 30 02:09:18.462628 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6eeacbd0-665d-4ea0-b13c-b1a4766be771 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:09:18.469726 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6b43bf2-3a8a-4b7d-b30e-4ed585d7cc7e admin admin] RBAC: Authorizing `identity:create_grant(project_id=7b8a54483ae04230ab5e0727c7eae0bc, user_id=c3557e6b6b2d46909db982ce76bbfdb8, role_id=1fb81afcb9f245f3b989add94598a063)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:09:18.472366 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6b43bf2-3a8a-4b7d-b30e-4ed585d7cc7e admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:09:18.473347 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6eeacbd0-665d-4ea0-b13c-b1a4766be771 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:09:18.478487 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5df25f0d-181e-4296-901f-69f87bb7049f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:09:18.479083 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5df25f0d-181e-4296-901f-69f87bb7049f admin admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:09:18.479365 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5df25f0d-181e-4296-901f-69f87bb7049f admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:09:18.479643 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5df25f0d-181e-4296-901f-69f87bb7049f admin admin] PATH_INFO: `/v3/roles` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:09:18.481038 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5df25f0d-181e-4296-901f-69f87bb7049f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:09:18.482761 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5df25f0d-181e-4296-901f-69f87bb7049f admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:09:18.484886 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=88,keystone:DELETE=7,keystone:INSERT=5 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:09:18.493113 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1277/2553] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Aug 30 02:09:18 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 02:09:18.497150 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5df25f0d-181e-4296-901f-69f87bb7049f admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:09:18.506481 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5df25f0d-181e-4296-901f-69f87bb7049f admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:09:18.511040 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a8d9697-aa7d-41cd-a6ac-e67e04ba7d71 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:09:18.511716 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a8d9697-aa7d-41cd-a6ac-e67e04ba7d71 admin admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:09:18.512004 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a8d9697-aa7d-41cd-a6ac-e67e04ba7d71 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:09:18.512275 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a8d9697-aa7d-41cd-a6ac-e67e04ba7d71 admin admin] PATH_INFO: `/v3/roles` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:09:18.512893 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a8d9697-aa7d-41cd-a6ac-e67e04ba7d71 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:09:18.514579 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a8d9697-aa7d-41cd-a6ac-e67e04ba7d71 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:09:18.532122 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1278/2554] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Aug 30 02:09:18 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 30 02:09:18.536203 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a8d9697-aa7d-41cd-a6ac-e67e04ba7d71 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:09:18.546649 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a8d9697-aa7d-41cd-a6ac-e67e04ba7d71 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:09:18.554250 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e2f1d7e-119e-4bd1-9116-086ee0c53a6d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:09:18.554926 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e2f1d7e-119e-4bd1-9116-086ee0c53a6d admin admin] REQUEST_METHOD: `PUT` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:09:18.555294 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e2f1d7e-119e-4bd1-9116-086ee0c53a6d admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:09:18.555984 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e2f1d7e-119e-4bd1-9116-086ee0c53a6d admin admin] PATH_INFO: `/v3/projects/4ad011b42e8d45459d4db0bf7792e63c/users/f3aaeadce91540d3a60eb7dfdcad3f6f/roles/1fb81afcb9f245f3b989add94598a063` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:09:18.557164 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1277/2555] 10.222.0.27 () {60 vars in 1483 bytes} [Tue Aug 30 02:09:18 2022] PUT /identity/v3/projects/7b8a54483ae04230ab5e0727c7eae0bc/users/c3557e6b6b2d46909db982ce76bbfdb8/roles/1fb81afcb9f245f3b989add94598a063 => generated 0 bytes in 118 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 30 02:09:18.570558 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-534e6696-f06f-439d-9d08-8b240ae08f1b None None] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:09:18.571141 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-534e6696-f06f-439d-9d08-8b240ae08f1b None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:09:18.571518 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-534e6696-f06f-439d-9d08-8b240ae08f1b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:09:18.578260 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e2f1d7e-119e-4bd1-9116-086ee0c53a6d admin admin] RBAC: Authorizing `identity:create_grant(project_id=4ad011b42e8d45459d4db0bf7792e63c, user_id=f3aaeadce91540d3a60eb7dfdcad3f6f, role_id=1fb81afcb9f245f3b989add94598a063)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:09:18.596998 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e2f1d7e-119e-4bd1-9116-086ee0c53a6d admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:09:18.614963 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.auth.core [None req-534e6696-f06f-439d-9d08-8b240ae08f1b None None] MFA Rules not processed for user `c3557e6b6b2d46909db982ce76bbfdb8`. Rule list: `[]` (Enabled: `True`). {{(pid=81578) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:09:18.616346 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-534e6696-f06f-439d-9d08-8b240ae08f1b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:09:18.649891 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:09:18.684515 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1278/2556] 10.222.0.27 () {58 vars in 958 bytes} [Tue Aug 30 02:09:18 2022] POST /identity/v3/auth/tokens => generated 3246 bytes in 116 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 30 02:09:18.688976 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-534e6696-f06f-439d-9d08-8b240ae08f1b None None] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:09:18.750893 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-534e6696-f06f-439d-9d08-8b240ae08f1b None None] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:09:18.755437 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdca8521-f996-4e45-925a-e10b1350ec8c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:09:18.756027 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdca8521-f996-4e45-925a-e10b1350ec8c admin admin] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:09:18.756304 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdca8521-f996-4e45-925a-e10b1350ec8c admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:09:18.756569 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdca8521-f996-4e45-925a-e10b1350ec8c admin admin] PATH_INFO: `/v3/projects` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:09:18.757196 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cdca8521-f996-4e45-925a-e10b1350ec8c admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:09:18.759081 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cdca8521-f996-4e45-925a-e10b1350ec8c admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:09:18.767850 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1279/2557] 10.222.0.27 () {60 vars in 1483 bytes} [Tue Aug 30 02:09:18 2022] PUT /identity/v3/projects/4ad011b42e8d45459d4db0bf7792e63c/users/f3aaeadce91540d3a60eb7dfdcad3f6f/roles/1fb81afcb9f245f3b989add94598a063 => generated 0 bytes in 232 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 30 02:09:18.779205 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f87eec8f-5c44-4be8-b39c-b85e63c813a2 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:09:18.779584 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f87eec8f-5c44-4be8-b39c-b85e63c813a2 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:09:18.779859 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f87eec8f-5c44-4be8-b39c-b85e63c813a2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:09:18.807980 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-f87eec8f-5c44-4be8-b39c-b85e63c813a2 None None] MFA Rules not processed for user `f3aaeadce91540d3a60eb7dfdcad3f6f`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:09:18.809897 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-f87eec8f-5c44-4be8-b39c-b85e63c813a2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:09:18.853570 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1279/2558] 10.222.0.27 () {60 vars in 1153 bytes} [Tue Aug 30 02:09:18 2022] POST /identity/v3/projects => generated 373 bytes in 163 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 30 02:09:18.856279 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdca8521-f996-4e45-925a-e10b1350ec8c admin admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:09:18.885857 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1280/2559] 10.222.0.27 () {58 vars in 958 bytes} [Tue Aug 30 02:09:18 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 30 02:09:18.890811 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f87eec8f-5c44-4be8-b39c-b85e63c813a2 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:09:18.924451 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdca8521-f996-4e45-925a-e10b1350ec8c admin admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:09:18.936072 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7206c2b2-6913-4620-bb63-83959af2bf9f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:09:18.936689 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7206c2b2-6913-4620-bb63-83959af2bf9f admin admin] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:09:18.936968 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7206c2b2-6913-4620-bb63-83959af2bf9f admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:09:18.937235 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7206c2b2-6913-4620-bb63-83959af2bf9f admin admin] PATH_INFO: `/v3/users` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:09:18.937974 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7206c2b2-6913-4620-bb63-83959af2bf9f admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:09:18.939853 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7206c2b2-6913-4620-bb63-83959af2bf9f admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:09:18.963905 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f87eec8f-5c44-4be8-b39c-b85e63c813a2 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:09:18.969142 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-683d396a-637a-49c1-b42d-f150df963a38 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:09:18.969772 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-683d396a-637a-49c1-b42d-f150df963a38 admin admin] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:09:18.970188 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-683d396a-637a-49c1-b42d-f150df963a38 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:09:18.970458 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-683d396a-637a-49c1-b42d-f150df963a38 admin admin] PATH_INFO: `/v3/projects` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:09:18.971079 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-683d396a-637a-49c1-b42d-f150df963a38 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:09:18.972959 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-683d396a-637a-49c1-b42d-f150df963a38 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:09:19.047507 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1280/2560] 10.222.0.27 () {60 vars in 1144 bytes} [Tue Aug 30 02:09:18 2022] POST /identity/v3/users => generated 293 bytes in 192 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 30 02:09:19.051511 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7206c2b2-6913-4620-bb63-83959af2bf9f admin admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:09:19.062501 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7206c2b2-6913-4620-bb63-83959af2bf9f admin admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:09:19.067450 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-366b0d79-7dbd-46db-a829-a0ce2753678a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:09:19.068078 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-366b0d79-7dbd-46db-a829-a0ce2753678a admin admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:09:19.068353 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-366b0d79-7dbd-46db-a829-a0ce2753678a admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:09:19.068619 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-366b0d79-7dbd-46db-a829-a0ce2753678a admin admin] PATH_INFO: `/v3/roles` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:09:19.069255 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-366b0d79-7dbd-46db-a829-a0ce2753678a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:09:19.070888 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-366b0d79-7dbd-46db-a829-a0ce2753678a admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:09:19.078819 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1281/2561] 10.222.0.27 () {60 vars in 1153 bytes} [Tue Aug 30 02:09:18 2022] POST /identity/v3/projects => generated 379 bytes in 187 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 30 02:09:19.085963 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-683d396a-637a-49c1-b42d-f150df963a38 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:09:19.086189 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1281/2562] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Aug 30 02:09:19 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 30 02:09:19.087118 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-366b0d79-7dbd-46db-a829-a0ce2753678a admin admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:09:19.097296 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-366b0d79-7dbd-46db-a829-a0ce2753678a admin admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:09:19.130805 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:09:19.134262 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:09:19.151222 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-683d396a-637a-49c1-b42d-f150df963a38 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:09:19.157376 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3454377d-71b8-426d-abcb-5f4689433e4d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:09:19.159129 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e78b7cc4-8548-4303-a7dd-d6b6189c5e32 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:09:19.159272 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3454377d-71b8-426d-abcb-5f4689433e4d admin admin] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:09:19.159373 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e78b7cc4-8548-4303-a7dd-d6b6189c5e32 admin admin] REQUEST_METHOD: `PUT` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:09:19.159484 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3454377d-71b8-426d-abcb-5f4689433e4d admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:09:19.159587 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e78b7cc4-8548-4303-a7dd-d6b6189c5e32 admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:09:19.159721 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3454377d-71b8-426d-abcb-5f4689433e4d admin admin] PATH_INFO: `/v3/users` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:09:19.159837 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e78b7cc4-8548-4303-a7dd-d6b6189c5e32 admin admin] PATH_INFO: `/v3/projects/2fa7ea3ec2ee47c2929f6f7172bb31e9/users/5e5c6d54442d45b589f7a8daf0997aa3/roles/ead5fac86a0747ea8823a82efcea8ed7` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:09:19.159935 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3454377d-71b8-426d-abcb-5f4689433e4d admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:09:19.161259 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3454377d-71b8-426d-abcb-5f4689433e4d admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:09:19.174688 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e78b7cc4-8548-4303-a7dd-d6b6189c5e32 admin admin] RBAC: Authorizing `identity:create_grant(project_id=2fa7ea3ec2ee47c2929f6f7172bb31e9, user_id=5e5c6d54442d45b589f7a8daf0997aa3, role_id=ead5fac86a0747ea8823a82efcea8ed7)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:09:19.177418 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e78b7cc4-8548-4303-a7dd-d6b6189c5e32 admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:09:19.281638 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1282/2563] 10.222.0.27 () {60 vars in 1144 bytes} [Tue Aug 30 02:09:19 2022] POST /identity/v3/users => generated 290 bytes in 200 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 02:09:19.286098 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3454377d-71b8-426d-abcb-5f4689433e4d admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:09:19.296201 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3454377d-71b8-426d-abcb-5f4689433e4d admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:09:19.300851 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7739225e-b901-41f7-8bb8-fbc0ac273d33 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:09:19.301558 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7739225e-b901-41f7-8bb8-fbc0ac273d33 admin admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:09:19.301891 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7739225e-b901-41f7-8bb8-fbc0ac273d33 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:09:19.302506 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7739225e-b901-41f7-8bb8-fbc0ac273d33 admin admin] PATH_INFO: `/v3/roles` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:09:19.303127 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7739225e-b901-41f7-8bb8-fbc0ac273d33 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:09:19.304745 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7739225e-b901-41f7-8bb8-fbc0ac273d33 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:09:19.310113 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1282/2564] 10.222.0.27 () {60 vars in 1483 bytes} [Tue Aug 30 02:09:19 2022] PUT /identity/v3/projects/2fa7ea3ec2ee47c2929f6f7172bb31e9/users/5e5c6d54442d45b589f7a8daf0997aa3/roles/ead5fac86a0747ea8823a82efcea8ed7 => generated 0 bytes in 223 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 02:09:19.314093 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e78b7cc4-8548-4303-a7dd-d6b6189c5e32 admin admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:09:19.321224 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1283/2565] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Aug 30 02:09:19 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 30 02:09:19.322304 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7739225e-b901-41f7-8bb8-fbc0ac273d33 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:09:19.337957 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7739225e-b901-41f7-8bb8-fbc0ac273d33 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:09:19.380688 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e78b7cc4-8548-4303-a7dd-d6b6189c5e32 admin admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:09:19.388892 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46017c42-97f6-4c5a-8ad3-388b67ef2df9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:09:19.388892 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46017c42-97f6-4c5a-8ad3-388b67ef2df9 admin admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:09:19.388892 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46017c42-97f6-4c5a-8ad3-388b67ef2df9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:09:19.388892 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46017c42-97f6-4c5a-8ad3-388b67ef2df9 admin admin] PATH_INFO: `/v3/roles` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:09:19.389555 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46017c42-97f6-4c5a-8ad3-388b67ef2df9 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:09:19.390082 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46017c42-97f6-4c5a-8ad3-388b67ef2df9 admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:09:19.404364 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1283/2566] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Aug 30 02:09:19 2022] GET /identity/v3/roles => generated 1420 bytes in 87 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 30 02:09:19.404610 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcc8024e-70f6-4617-88d1-91549e735fff admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:09:19.404836 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcc8024e-70f6-4617-88d1-91549e735fff admin admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:09:19.405131 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcc8024e-70f6-4617-88d1-91549e735fff admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:09:19.405482 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcc8024e-70f6-4617-88d1-91549e735fff admin admin] PATH_INFO: `/v3/roles` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:09:19.405999 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46017c42-97f6-4c5a-8ad3-388b67ef2df9 admin admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:09:19.406216 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bcc8024e-70f6-4617-88d1-91549e735fff admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:09:19.407872 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bcc8024e-70f6-4617-88d1-91549e735fff admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:09:19.416115 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46017c42-97f6-4c5a-8ad3-388b67ef2df9 admin admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:09:19.423719 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1284/2567] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Aug 30 02:09:19 2022] GET /identity/v3/roles => generated 1420 bytes in 99 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 02:09:19.424869 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcc8024e-70f6-4617-88d1-91549e735fff admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:09:19.426883 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-504327b1-33a7-418f-aae5-11fffe667928 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:09:19.427664 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-504327b1-33a7-418f-aae5-11fffe667928 admin admin] REQUEST_METHOD: `PUT` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:09:19.428048 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-504327b1-33a7-418f-aae5-11fffe667928 admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:09:19.428420 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-504327b1-33a7-418f-aae5-11fffe667928 admin admin] PATH_INFO: `/v3/projects/2fa7ea3ec2ee47c2929f6f7172bb31e9/users/5e5c6d54442d45b589f7a8daf0997aa3/roles/ead5fac86a0747ea8823a82efcea8ed7` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:09:19.430533 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-504327b1-33a7-418f-aae5-11fffe667928 admin admin] RBAC: Authorizing `identity:create_grant(project_id=2fa7ea3ec2ee47c2929f6f7172bb31e9, user_id=5e5c6d54442d45b589f7a8daf0997aa3, role_id=ead5fac86a0747ea8823a82efcea8ed7)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:09:19.433253 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-504327b1-33a7-418f-aae5-11fffe667928 admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:09:19.439075 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcc8024e-70f6-4617-88d1-91549e735fff admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:09:19.458488 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1284/2568] 10.222.0.27 () {60 vars in 1483 bytes} [Tue Aug 30 02:09:19 2022] PUT /identity/v3/projects/2fa7ea3ec2ee47c2929f6f7172bb31e9/users/5e5c6d54442d45b589f7a8daf0997aa3/roles/ead5fac86a0747ea8823a82efcea8ed7 => generated 0 bytes in 42 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 02:09:19.459427 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26b48020-6e64-4fb1-9c82-5ea8889df03b None None] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:09:19.459714 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26b48020-6e64-4fb1-9c82-5ea8889df03b None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:09:19.460069 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26b48020-6e64-4fb1-9c82-5ea8889df03b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:09:19.497207 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.auth.core [None req-26b48020-6e64-4fb1-9c82-5ea8889df03b None None] MFA Rules not processed for user `5e5c6d54442d45b589f7a8daf0997aa3`. Rule list: `[]` (Enabled: `True`). {{(pid=81578) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:09:19.498543 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-26b48020-6e64-4fb1-9c82-5ea8889df03b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:09:19.522264 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f82bf36a-493c-4334-ab28-d519f4b543a5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:09:19.522951 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f82bf36a-493c-4334-ab28-d519f4b543a5 admin admin] REQUEST_METHOD: `PUT` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:09:19.523238 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f82bf36a-493c-4334-ab28-d519f4b543a5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:09:19.523576 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f82bf36a-493c-4334-ab28-d519f4b543a5 admin admin] PATH_INFO: `/v3/projects/5d31acae43e74479a3e0ff4ebbf539bb/users/d2c4e1d3ba69433b843e9a5cc79be2da/roles/1fb81afcb9f245f3b989add94598a063` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:09:19.539347 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f82bf36a-493c-4334-ab28-d519f4b543a5 admin admin] RBAC: Authorizing `identity:create_grant(project_id=5d31acae43e74479a3e0ff4ebbf539bb, user_id=d2c4e1d3ba69433b843e9a5cc79be2da, role_id=1fb81afcb9f245f3b989add94598a063)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:09:19.542446 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f82bf36a-493c-4334-ab28-d519f4b543a5 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:09:19.568553 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1285/2569] 10.222.0.27 () {58 vars in 958 bytes} [Tue Aug 30 02:09:19 2022] POST /identity/v3/auth/tokens => generated 3317 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 02:09:19.580293 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26b48020-6e64-4fb1-9c82-5ea8889df03b None None] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:09:19.581646 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-26b48020-6e64-4fb1-9c82-5ea8889df03b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:09:19.611929 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:09:19.616318 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1285/2570] 10.222.0.27 () {60 vars in 1483 bytes} [Tue Aug 30 02:09:19 2022] PUT /identity/v3/projects/5d31acae43e74479a3e0ff4ebbf539bb/users/d2c4e1d3ba69433b843e9a5cc79be2da/roles/1fb81afcb9f245f3b989add94598a063 => generated 0 bytes in 192 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 02:09:19.626970 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e31328d0-c3fa-4857-91eb-05f6cc6704ad None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:09:19.627262 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e31328d0-c3fa-4857-91eb-05f6cc6704ad None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:09:19.627542 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e31328d0-c3fa-4857-91eb-05f6cc6704ad None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:09:19.648010 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26b48020-6e64-4fb1-9c82-5ea8889df03b None None] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:09:19.657395 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-e31328d0-c3fa-4857-91eb-05f6cc6704ad None None] MFA Rules not processed for user `d2c4e1d3ba69433b843e9a5cc79be2da`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:09:19.658655 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-e31328d0-c3fa-4857-91eb-05f6cc6704ad None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:09:19.682829 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:09:19.689889 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f1af503-70a3-446e-9a53-07967009cebf service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ab936b3767a8417aa80c8c8f1b84b667', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4562f3951fbe465280073e60a11e44e2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:09:19.690569 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f1af503-70a3-446e-9a53-07967009cebf service cinder] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:09:19.690915 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f1af503-70a3-446e-9a53-07967009cebf service cinder] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:09:19.691339 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f1af503-70a3-446e-9a53-07967009cebf service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:09:19.729370 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1286/2571] 10.222.0.27 () {58 vars in 958 bytes} [Tue Aug 30 02:09:19 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 30 02:09:19.733959 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e31328d0-c3fa-4857-91eb-05f6cc6704ad None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:09:19.754878 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f1af503-70a3-446e-9a53-07967009cebf service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:09:19.756294 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f1af503-70a3-446e-9a53-07967009cebf service cinder] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:09:19.766659 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1286/2572] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Aug 30 02:09:19 2022] GET /identity/v3/auth/tokens => generated 3317 bytes in 187 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 30 02:09:19.798065 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e31328d0-c3fa-4857-91eb-05f6cc6704ad None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:09:19.802867 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a45619d9-66f5-4bba-8751-d7509db6f5fb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:09:19.803471 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a45619d9-66f5-4bba-8751-d7509db6f5fb admin admin] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:09:19.803750 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a45619d9-66f5-4bba-8751-d7509db6f5fb admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:09:19.804021 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a45619d9-66f5-4bba-8751-d7509db6f5fb admin admin] PATH_INFO: `/v3/projects` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:09:19.804689 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a45619d9-66f5-4bba-8751-d7509db6f5fb admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:09:19.806789 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a45619d9-66f5-4bba-8751-d7509db6f5fb admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:09:19.928243 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1287/2573] 10.222.0.27 () {60 vars in 1153 bytes} [Tue Aug 30 02:09:19 2022] POST /identity/v3/projects => generated 379 bytes in 195 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 30 02:09:19.932327 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f1af503-70a3-446e-9a53-07967009cebf service cinder] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:09:19.943361 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f1af503-70a3-446e-9a53-07967009cebf service cinder] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:09:19.970087 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a45619d9-66f5-4bba-8751-d7509db6f5fb admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:09:20.002312 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22f500cf-4960-479b-a99b-c41d7df10bd9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:09:20.002972 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22f500cf-4960-479b-a99b-c41d7df10bd9 admin admin] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:09:20.003373 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22f500cf-4960-479b-a99b-c41d7df10bd9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:09:20.003649 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22f500cf-4960-479b-a99b-c41d7df10bd9 admin admin] PATH_INFO: `/v3/users` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:09:20.004277 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22f500cf-4960-479b-a99b-c41d7df10bd9 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:09:20.006209 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22f500cf-4960-479b-a99b-c41d7df10bd9 admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:09:20.027962 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a45619d9-66f5-4bba-8751-d7509db6f5fb admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:09:20.031808 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-988fb5dc-1606-46fc-bd61-0e56617c451e service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ab936b3767a8417aa80c8c8f1b84b667', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4562f3951fbe465280073e60a11e44e2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:09:20.032415 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-988fb5dc-1606-46fc-bd61-0e56617c451e service cinder] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:09:20.032687 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-988fb5dc-1606-46fc-bd61-0e56617c451e service cinder] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:09:20.032962 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-988fb5dc-1606-46fc-bd61-0e56617c451e service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:09:20.079265 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1287/2574] 10.222.0.27 () {60 vars in 1144 bytes} [Tue Aug 30 02:09:19 2022] POST /identity/v3/users => generated 296 bytes in 144 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 02:09:20.080637 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22f500cf-4960-479b-a99b-c41d7df10bd9 admin admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:09:20.090164 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-988fb5dc-1606-46fc-bd61-0e56617c451e service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:09:20.091627 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-988fb5dc-1606-46fc-bd61-0e56617c451e service cinder] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:09:20.091872 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22f500cf-4960-479b-a99b-c41d7df10bd9 admin admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:09:20.096626 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3c6e999-958f-4992-ac74-2c4e9d28910e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:09:20.097073 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3c6e999-958f-4992-ac74-2c4e9d28910e admin admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:09:20.097342 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3c6e999-958f-4992-ac74-2c4e9d28910e admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:09:20.097708 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3c6e999-958f-4992-ac74-2c4e9d28910e admin admin] PATH_INFO: `/v3/roles` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:09:20.104808 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3c6e999-958f-4992-ac74-2c4e9d28910e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:09:20.104808 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3c6e999-958f-4992-ac74-2c4e9d28910e admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:09:20.105056 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1288/2575] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Aug 30 02:09:19 2022] GET /identity/v3/auth/tokens => generated 3246 bytes in 131 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 30 02:09:20.114318 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1288/2576] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Aug 30 02:09:20 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 02:09:20.115721 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-988fb5dc-1606-46fc-bd61-0e56617c451e service cinder] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:09:20.123041 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:09:20.127252 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-988fb5dc-1606-46fc-bd61-0e56617c451e service cinder] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:09:20.131835 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96259ccf-fcfc-47ed-8226-20bfd66f5332 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:09:20.132544 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96259ccf-fcfc-47ed-8226-20bfd66f5332 admin admin] REQUEST_METHOD: `PUT` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:09:20.132826 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96259ccf-fcfc-47ed-8226-20bfd66f5332 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:09:20.133101 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96259ccf-fcfc-47ed-8226-20bfd66f5332 admin admin] PATH_INFO: `/v3/projects/b69d4bab45b949dea2d3c67ef969c868/users/a3e6c917504943d59a8f33dfdd104423/roles/ead5fac86a0747ea8823a82efcea8ed7` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:09:20.148626 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96259ccf-fcfc-47ed-8226-20bfd66f5332 admin admin] RBAC: Authorizing `identity:create_grant(project_id=b69d4bab45b949dea2d3c67ef969c868, user_id=a3e6c917504943d59a8f33dfdd104423, role_id=ead5fac86a0747ea8823a82efcea8ed7)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:09:20.151299 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96259ccf-fcfc-47ed-8226-20bfd66f5332 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:09:20.256156 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1289/2577] 10.222.0.27 () {60 vars in 1483 bytes} [Tue Aug 30 02:09:20 2022] PUT /identity/v3/projects/b69d4bab45b949dea2d3c67ef969c868/users/a3e6c917504943d59a8f33dfdd104423/roles/ead5fac86a0747ea8823a82efcea8ed7 => generated 0 bytes in 141 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 02:09:20.260260 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3c6e999-958f-4992-ac74-2c4e9d28910e admin admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:09:20.270741 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3c6e999-958f-4992-ac74-2c4e9d28910e admin admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:09:20.278488 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:09:20.328397 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bed6aeee-ab3c-472e-afe9-dd41dd44fd4c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:09:20.329785 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bed6aeee-ab3c-472e-afe9-dd41dd44fd4c admin admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:09:20.329785 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bed6aeee-ab3c-472e-afe9-dd41dd44fd4c admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:09:20.329785 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bed6aeee-ab3c-472e-afe9-dd41dd44fd4c admin admin] PATH_INFO: `/v3/roles` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:09:20.330128 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bed6aeee-ab3c-472e-afe9-dd41dd44fd4c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:09:20.331509 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bed6aeee-ab3c-472e-afe9-dd41dd44fd4c admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:09:20.341720 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1289/2578] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Aug 30 02:09:20 2022] GET /identity/v3/roles => generated 1420 bytes in 82 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 02:09:20.345926 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96259ccf-fcfc-47ed-8226-20bfd66f5332 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:09:20.356861 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96259ccf-fcfc-47ed-8226-20bfd66f5332 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:09:20.361470 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7339908-b5fc-456a-a29d-40e1830a56ff admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:09:20.362117 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7339908-b5fc-456a-a29d-40e1830a56ff admin admin] REQUEST_METHOD: `PUT` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:09:20.362397 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7339908-b5fc-456a-a29d-40e1830a56ff admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:09:20.362668 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7339908-b5fc-456a-a29d-40e1830a56ff admin admin] PATH_INFO: `/v3/projects/b69d4bab45b949dea2d3c67ef969c868/users/a3e6c917504943d59a8f33dfdd104423/roles/ead5fac86a0747ea8823a82efcea8ed7` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:09:20.364595 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7339908-b5fc-456a-a29d-40e1830a56ff admin admin] RBAC: Authorizing `identity:create_grant(project_id=b69d4bab45b949dea2d3c67ef969c868, user_id=a3e6c917504943d59a8f33dfdd104423, role_id=ead5fac86a0747ea8823a82efcea8ed7)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:09:20.367250 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7339908-b5fc-456a-a29d-40e1830a56ff admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:09:20.378066 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1290/2579] 10.222.0.27 () {60 vars in 1483 bytes} [Tue Aug 30 02:09:20 2022] PUT /identity/v3/projects/b69d4bab45b949dea2d3c67ef969c868/users/a3e6c917504943d59a8f33dfdd104423/roles/ead5fac86a0747ea8823a82efcea8ed7 => generated 0 bytes in 33 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 02:09:20.389233 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e8529b8-3dd3-48f1-97ab-f5f827751ce0 None None] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:09:20.390009 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e8529b8-3dd3-48f1-97ab-f5f827751ce0 None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:09:20.390009 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e8529b8-3dd3-48f1-97ab-f5f827751ce0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:09:20.420379 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.auth.core [None req-2e8529b8-3dd3-48f1-97ab-f5f827751ce0 None None] MFA Rules not processed for user `a3e6c917504943d59a8f33dfdd104423`. Rule list: `[]` (Enabled: `True`). {{(pid=81578) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:09:20.421646 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-2e8529b8-3dd3-48f1-97ab-f5f827751ce0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:09:20.489642 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1290/2580] 10.222.0.27 () {58 vars in 958 bytes} [Tue Aug 30 02:09:20 2022] POST /identity/v3/auth/tokens => generated 3323 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 02:09:20.502118 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7339908-b5fc-456a-a29d-40e1830a56ff admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:09:20.503792 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-d7339908-b5fc-456a-a29d-40e1830a56ff admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:09:20.586467 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7339908-b5fc-456a-a29d-40e1830a56ff admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:09:20.590456 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7607034-6b84-467d-91c8-5263ac919109 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ab936b3767a8417aa80c8c8f1b84b667', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4562f3951fbe465280073e60a11e44e2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:09:20.591128 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7607034-6b84-467d-91c8-5263ac919109 service cinder] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:09:20.591516 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7607034-6b84-467d-91c8-5263ac919109 service cinder] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:09:20.591734 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7607034-6b84-467d-91c8-5263ac919109 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:09:20.601520 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7607034-6b84-467d-91c8-5263ac919109 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:09:20.603311 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7607034-6b84-467d-91c8-5263ac919109 service cinder] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:09:20.613894 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1291/2581] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Aug 30 02:09:20 2022] GET /identity/v3/auth/tokens => generated 3323 bytes in 112 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 02:09:20.718959 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e8529b8-3dd3-48f1-97ab-f5f827751ce0 None None] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:09:20.729176 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e8529b8-3dd3-48f1-97ab-f5f827751ce0 None None] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:09:20.733793 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba12c42b-cc0d-468d-85f3-019a7fa89f06 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ab936b3767a8417aa80c8c8f1b84b667', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4562f3951fbe465280073e60a11e44e2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:09:20.736689 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba12c42b-cc0d-468d-85f3-019a7fa89f06 service cinder] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:09:20.736689 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba12c42b-cc0d-468d-85f3-019a7fa89f06 service cinder] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:09:20.736689 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba12c42b-cc0d-468d-85f3-019a7fa89f06 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:09:20.792584 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba12c42b-cc0d-468d-85f3-019a7fa89f06 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:09:20.795126 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba12c42b-cc0d-468d-85f3-019a7fa89f06 service cinder] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:09:20.804346 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1291/2582] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Aug 30 02:09:20 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 86 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 02:09:21.779559 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b24bba9b-c04f-4d69-a429-1066fd8d89b8 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:09:21.783082 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b24bba9b-c04f-4d69-a429-1066fd8d89b8 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:09:21.783082 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b24bba9b-c04f-4d69-a429-1066fd8d89b8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:09:21.801922 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-b24bba9b-c04f-4d69-a429-1066fd8d89b8 None None] MFA Rules not processed for user `121c374d49a64335ab74a431cfbc1320`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:09:21.803130 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-b24bba9b-c04f-4d69-a429-1066fd8d89b8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:09:21.874221 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1292/2583] 10.222.0.27 () {58 vars in 1009 bytes} [Tue Aug 30 02:09:21 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 02:09:22.014503 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba12c42b-cc0d-468d-85f3-019a7fa89f06 service cinder] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:09:22.015946 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-ba12c42b-cc0d-468d-85f3-019a7fa89f06 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:09:22.050483 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:09:22.087556 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba12c42b-cc0d-468d-85f3-019a7fa89f06 service cinder] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:09:22.093622 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13a5fea2-1ecb-4a56-b70f-078cd9819a11 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b4f0008fde424f22bc1f7307cf422774', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4562f3951fbe465280073e60a11e44e2', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:09:22.094228 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13a5fea2-1ecb-4a56-b70f-078cd9819a11 service swift] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:09:22.094509 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13a5fea2-1ecb-4a56-b70f-078cd9819a11 service swift] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:09:22.094782 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13a5fea2-1ecb-4a56-b70f-078cd9819a11 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:09:22.103304 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13a5fea2-1ecb-4a56-b70f-078cd9819a11 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:09:22.104926 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13a5fea2-1ecb-4a56-b70f-078cd9819a11 service swift] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:09:22.113708 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1292/2584] 10.222.0.27 () {60 vars in 1384 bytes} [Tue Aug 30 02:09:22 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 100 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Aug 30 02:09:26.207236 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b24bba9b-c04f-4d69-a429-1066fd8d89b8 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:09:26.284090 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b24bba9b-c04f-4d69-a429-1066fd8d89b8 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:09:26.290055 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49c89b7d-14c9-4dd8-a05f-6355477fdf41 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:09:26.290698 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49c89b7d-14c9-4dd8-a05f-6355477fdf41 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:09:26.290974 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49c89b7d-14c9-4dd8-a05f-6355477fdf41 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:09:26.291242 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49c89b7d-14c9-4dd8-a05f-6355477fdf41 admin admin] PATH_INFO: `/v3/users/f3aaeadce91540d3a60eb7dfdcad3f6f` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:09:26.292375 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49c89b7d-14c9-4dd8-a05f-6355477fdf41 admin admin] RBAC: Authorizing `identity:delete_user(user_id=f3aaeadce91540d3a60eb7dfdcad3f6f)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:09:26.294965 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49c89b7d-14c9-4dd8-a05f-6355477fdf41 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:09:26.314539 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=97,keystone:INSERT=2,keystone:DELETE=1 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:09:26.504709 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-49c89b7d-14c9-4dd8-a05f-6355477fdf41 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'f3aaeadce91540d3a60eb7dfdcad3f6f'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:09:26.505403 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-49c89b7d-14c9-4dd8-a05f-6355477fdf41 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'f3aaeadce91540d3a60eb7dfdcad3f6f'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:09:26.506021 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-49c89b7d-14c9-4dd8-a05f-6355477fdf41 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'f3aaeadce91540d3a60eb7dfdcad3f6f'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:09:26.522108 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-49c89b7d-14c9-4dd8-a05f-6355477fdf41 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'f3aaeadce91540d3a60eb7dfdcad3f6f'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:09:26.533406 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-49c89b7d-14c9-4dd8-a05f-6355477fdf41 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'f3aaeadce91540d3a60eb7dfdcad3f6f'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:09:26.601820 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1293/2585] 10.222.0.27 () {58 vars in 1224 bytes} [Tue Aug 30 02:09:26 2022] DELETE /identity/v3/users/f3aaeadce91540d3a60eb7dfdcad3f6f => generated 0 bytes in 397 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 02:09:26.609843 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cfdf5db-696d-4c22-ad8a-3451b0e87c93 None None] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:09:26.610381 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cfdf5db-696d-4c22-ad8a-3451b0e87c93 None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:09:26.610381 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cfdf5db-696d-4c22-ad8a-3451b0e87c93 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:09:26.635367 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.auth.core [None req-6cfdf5db-696d-4c22-ad8a-3451b0e87c93 None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81578) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:09:26.636554 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-6cfdf5db-696d-4c22-ad8a-3451b0e87c93 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:09:26.714678 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1293/2586] 10.222.0.27 () {58 vars in 958 bytes} [Tue Aug 30 02:09:26 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 02:09:26.726667 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49c89b7d-14c9-4dd8-a05f-6355477fdf41 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:09:26.728618 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-49c89b7d-14c9-4dd8-a05f-6355477fdf41 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:09:26.804992 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49c89b7d-14c9-4dd8-a05f-6355477fdf41 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:09:26.809389 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a28bd452-b418-4268-83a1-9edb7f7dba59 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f3ea1580c56f4814a92bbb2acf542905', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4562f3951fbe465280073e60a11e44e2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:09:26.810355 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a28bd452-b418-4268-83a1-9edb7f7dba59 service neutron] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:09:26.810773 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a28bd452-b418-4268-83a1-9edb7f7dba59 service neutron] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:09:26.811835 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a28bd452-b418-4268-83a1-9edb7f7dba59 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:09:26.826437 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a28bd452-b418-4268-83a1-9edb7f7dba59 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:09:26.827994 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a28bd452-b418-4268-83a1-9edb7f7dba59 service neutron] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:09:26.843354 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1294/2587] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Aug 30 02:09:26 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 117 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 02:09:27.311753 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cfdf5db-696d-4c22-ad8a-3451b0e87c93 None None] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:09:27.313558 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-6cfdf5db-696d-4c22-ad8a-3451b0e87c93 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:09:27.330143 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cfdf5db-696d-4c22-ad8a-3451b0e87c93 None None] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:09:27.337959 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb758e23-04b0-4107-a060-51bfc23084e8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:09:27.337959 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb758e23-04b0-4107-a060-51bfc23084e8 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:09:27.337959 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb758e23-04b0-4107-a060-51bfc23084e8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:09:27.337959 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb758e23-04b0-4107-a060-51bfc23084e8 admin admin] PATH_INFO: `/v3/projects/4ad011b42e8d45459d4db0bf7792e63c` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:09:27.351829 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb758e23-04b0-4107-a060-51bfc23084e8 admin admin] RBAC: Authorizing `identity:delete_project(project_id=4ad011b42e8d45459d4db0bf7792e63c)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:09:27.354824 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb758e23-04b0-4107-a060-51bfc23084e8 admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:09:27.447310 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-eb758e23-04b0-4107-a060-51bfc23084e8 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '4ad011b42e8d45459d4db0bf7792e63c', 'request_id': 'req-eb758e23-04b0-4107-a060-51bfc23084e8'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:09:27.454431 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-eb758e23-04b0-4107-a060-51bfc23084e8 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '4ad011b42e8d45459d4db0bf7792e63c', 'request_id': 'req-eb758e23-04b0-4107-a060-51bfc23084e8'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:09:27.462178 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-eb758e23-04b0-4107-a060-51bfc23084e8 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '4ad011b42e8d45459d4db0bf7792e63c', 'request_id': 'req-eb758e23-04b0-4107-a060-51bfc23084e8'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:09:27.551101 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-eb758e23-04b0-4107-a060-51bfc23084e8 admin admin] The token cache is being invalidate because project 4ad011b42e8d45459d4db0bf7792e63c was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81578) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 30 02:09:27.551400 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-eb758e23-04b0-4107-a060-51bfc23084e8 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:09:27.552023 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-eb758e23-04b0-4107-a060-51bfc23084e8 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:09:27.554057 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1294/2588] 10.222.0.27 () {58 vars in 1233 bytes} [Tue Aug 30 02:09:27 2022] DELETE /identity/v3/projects/4ad011b42e8d45459d4db0bf7792e63c => generated 0 bytes in 243 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 02:09:27.561142 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a28bd452-b418-4268-83a1-9edb7f7dba59 service neutron] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:09:27.572531 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a28bd452-b418-4268-83a1-9edb7f7dba59 service neutron] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:09:27.657309 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62fdd91b-aefe-4c07-9f09-b37fbe75fc24 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:09:27.658005 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62fdd91b-aefe-4c07-9f09-b37fbe75fc24 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:09:27.658282 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62fdd91b-aefe-4c07-9f09-b37fbe75fc24 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:09:27.658554 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62fdd91b-aefe-4c07-9f09-b37fbe75fc24 admin admin] PATH_INFO: `/v3/users/d2c4e1d3ba69433b843e9a5cc79be2da` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:09:27.659797 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62fdd91b-aefe-4c07-9f09-b37fbe75fc24 admin admin] RBAC: Authorizing `identity:delete_user(user_id=d2c4e1d3ba69433b843e9a5cc79be2da)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:09:27.661893 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62fdd91b-aefe-4c07-9f09-b37fbe75fc24 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:09:28.293529 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:DELETE=14,keystone:SELECT=85,keystone:INSERT=1 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:09:28.299316 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-62fdd91b-aefe-4c07-9f09-b37fbe75fc24 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'd2c4e1d3ba69433b843e9a5cc79be2da'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:09:28.299979 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-62fdd91b-aefe-4c07-9f09-b37fbe75fc24 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'd2c4e1d3ba69433b843e9a5cc79be2da'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:09:28.300589 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-62fdd91b-aefe-4c07-9f09-b37fbe75fc24 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'd2c4e1d3ba69433b843e9a5cc79be2da'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:09:28.313026 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-62fdd91b-aefe-4c07-9f09-b37fbe75fc24 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'd2c4e1d3ba69433b843e9a5cc79be2da'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:09:28.322169 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-62fdd91b-aefe-4c07-9f09-b37fbe75fc24 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'd2c4e1d3ba69433b843e9a5cc79be2da'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:09:28.421704 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1295/2589] 10.222.0.27 () {58 vars in 1224 bytes} [Tue Aug 30 02:09:27 2022] DELETE /identity/v3/users/d2c4e1d3ba69433b843e9a5cc79be2da => generated 0 bytes in 864 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 30 02:09:28.924290 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb758e23-04b0-4107-a060-51bfc23084e8 admin admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:09:28.926750 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-eb758e23-04b0-4107-a060-51bfc23084e8 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:09:28.960768 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb758e23-04b0-4107-a060-51bfc23084e8 admin admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:09:29.026931 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:09:29.045421 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9716909-d0cc-4790-9e4a-6f9fa5f439a8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:09:29.046103 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9716909-d0cc-4790-9e4a-6f9fa5f439a8 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:09:29.046363 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9716909-d0cc-4790-9e4a-6f9fa5f439a8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:09:29.046630 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9716909-d0cc-4790-9e4a-6f9fa5f439a8 admin admin] PATH_INFO: `/v3/projects/5d31acae43e74479a3e0ff4ebbf539bb` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:09:29.049506 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9716909-d0cc-4790-9e4a-6f9fa5f439a8 admin admin] RBAC: Authorizing `identity:delete_project(project_id=5d31acae43e74479a3e0ff4ebbf539bb)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:09:29.049937 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9716909-d0cc-4790-9e4a-6f9fa5f439a8 admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:09:29.206379 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-f9716909-d0cc-4790-9e4a-6f9fa5f439a8 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '5d31acae43e74479a3e0ff4ebbf539bb', 'request_id': 'req-f9716909-d0cc-4790-9e4a-6f9fa5f439a8'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:09:29.217000 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-f9716909-d0cc-4790-9e4a-6f9fa5f439a8 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '5d31acae43e74479a3e0ff4ebbf539bb', 'request_id': 'req-f9716909-d0cc-4790-9e4a-6f9fa5f439a8'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:09:29.225845 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-f9716909-d0cc-4790-9e4a-6f9fa5f439a8 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '5d31acae43e74479a3e0ff4ebbf539bb', 'request_id': 'req-f9716909-d0cc-4790-9e4a-6f9fa5f439a8'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:09:29.304873 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-f9716909-d0cc-4790-9e4a-6f9fa5f439a8 admin admin] The token cache is being invalidate because project 5d31acae43e74479a3e0ff4ebbf539bb was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81578) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 30 02:09:29.305246 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-f9716909-d0cc-4790-9e4a-6f9fa5f439a8 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:09:29.306056 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-f9716909-d0cc-4790-9e4a-6f9fa5f439a8 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:09:29.307612 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1295/2590] 10.222.0.27 () {58 vars in 1233 bytes} [Tue Aug 30 02:09:28 2022] DELETE /identity/v3/projects/5d31acae43e74479a3e0ff4ebbf539bb => generated 0 bytes in 384 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 02:09:29.312577 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62fdd91b-aefe-4c07-9f09-b37fbe75fc24 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:09:29.314682 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-62fdd91b-aefe-4c07-9f09-b37fbe75fc24 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:09:29.329351 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62fdd91b-aefe-4c07-9f09-b37fbe75fc24 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:09:29.416331 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea954706-ef60-42a9-81af-2543731b0fb7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:09:29.421628 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea954706-ef60-42a9-81af-2543731b0fb7 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:09:29.421928 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea954706-ef60-42a9-81af-2543731b0fb7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:09:29.422201 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea954706-ef60-42a9-81af-2543731b0fb7 admin admin] PATH_INFO: `/v3/users/a3e6c917504943d59a8f33dfdd104423` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:09:29.423489 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea954706-ef60-42a9-81af-2543731b0fb7 admin admin] RBAC: Authorizing `identity:delete_user(user_id=a3e6c917504943d59a8f33dfdd104423)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:09:29.430069 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea954706-ef60-42a9-81af-2543731b0fb7 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:09:29.625282 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-ea954706-ef60-42a9-81af-2543731b0fb7 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'a3e6c917504943d59a8f33dfdd104423'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:09:29.625987 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-ea954706-ef60-42a9-81af-2543731b0fb7 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'a3e6c917504943d59a8f33dfdd104423'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:09:29.626789 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-ea954706-ef60-42a9-81af-2543731b0fb7 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'a3e6c917504943d59a8f33dfdd104423'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:09:29.641807 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-ea954706-ef60-42a9-81af-2543731b0fb7 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'a3e6c917504943d59a8f33dfdd104423'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:09:29.650386 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-ea954706-ef60-42a9-81af-2543731b0fb7 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'a3e6c917504943d59a8f33dfdd104423'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:09:29.740377 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1296/2591] 10.222.0.27 () {58 vars in 1224 bytes} [Tue Aug 30 02:09:29 2022] DELETE /identity/v3/users/a3e6c917504943d59a8f33dfdd104423 => generated 0 bytes in 428 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 02:09:30.263221 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9716909-d0cc-4790-9e4a-6f9fa5f439a8 admin admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:09:30.264849 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-f9716909-d0cc-4790-9e4a-6f9fa5f439a8 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:09:30.288665 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9716909-d0cc-4790-9e4a-6f9fa5f439a8 admin admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:09:30.445897 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d09c49ad-ecb8-42aa-8d7d-afe45e0f9202 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:09:30.446548 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d09c49ad-ecb8-42aa-8d7d-afe45e0f9202 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:09:30.446832 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d09c49ad-ecb8-42aa-8d7d-afe45e0f9202 admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:09:30.447108 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d09c49ad-ecb8-42aa-8d7d-afe45e0f9202 admin admin] PATH_INFO: `/v3/projects/b69d4bab45b949dea2d3c67ef969c868` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:09:30.448265 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d09c49ad-ecb8-42aa-8d7d-afe45e0f9202 admin admin] RBAC: Authorizing `identity:delete_project(project_id=b69d4bab45b949dea2d3c67ef969c868)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:09:30.451074 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d09c49ad-ecb8-42aa-8d7d-afe45e0f9202 admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:09:30.611715 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-d09c49ad-ecb8-42aa-8d7d-afe45e0f9202 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'b69d4bab45b949dea2d3c67ef969c868', 'request_id': 'req-d09c49ad-ecb8-42aa-8d7d-afe45e0f9202'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:09:30.618963 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-d09c49ad-ecb8-42aa-8d7d-afe45e0f9202 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'b69d4bab45b949dea2d3c67ef969c868', 'request_id': 'req-d09c49ad-ecb8-42aa-8d7d-afe45e0f9202'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:09:30.627022 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-d09c49ad-ecb8-42aa-8d7d-afe45e0f9202 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'b69d4bab45b949dea2d3c67ef969c868', 'request_id': 'req-d09c49ad-ecb8-42aa-8d7d-afe45e0f9202'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:09:30.709678 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-d09c49ad-ecb8-42aa-8d7d-afe45e0f9202 admin admin] The token cache is being invalidate because project b69d4bab45b949dea2d3c67ef969c868 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81578) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 30 02:09:30.710235 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-d09c49ad-ecb8-42aa-8d7d-afe45e0f9202 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:09:30.711183 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-d09c49ad-ecb8-42aa-8d7d-afe45e0f9202 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:09:30.713013 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1296/2592] 10.222.0.27 () {58 vars in 1233 bytes} [Tue Aug 30 02:09:30 2022] DELETE /identity/v3/projects/b69d4bab45b949dea2d3c67ef969c868 => generated 0 bytes in 451 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 02:09:30.720130 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1f7dff6-6fb0-47d4-a478-bb8a91fb3722 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:09:30.721218 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1f7dff6-6fb0-47d4-a478-bb8a91fb3722 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:09:30.721797 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1f7dff6-6fb0-47d4-a478-bb8a91fb3722 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:09:30.763608 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-c1f7dff6-6fb0-47d4-a478-bb8a91fb3722 None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:09:30.764735 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-c1f7dff6-6fb0-47d4-a478-bb8a91fb3722 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:09:30.804337 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:DELETE=15,keystone:SELECT=83,keystone:INSERT=2 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:09:30.839888 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1297/2593] 10.222.0.27 () {58 vars in 958 bytes} [Tue Aug 30 02:09:30 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 121 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 02:09:30.849669 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cbe68ed-4315-4342-a567-db150a19607c None None] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:09:30.850210 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cbe68ed-4315-4342-a567-db150a19607c None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:09:30.850555 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cbe68ed-4315-4342-a567-db150a19607c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:09:30.861749 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=84,keystone:DELETE=14,keystone:INSERT=2 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:09:30.878007 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.auth.core [None req-8cbe68ed-4315-4342-a567-db150a19607c None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81578) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:09:30.879734 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-8cbe68ed-4315-4342-a567-db150a19607c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:09:30.889348 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1297/2594] 10.222.0.27 () {58 vars in 958 bytes} [Tue Aug 30 02:09:30 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 41 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 02:09:30.893762 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1f7dff6-6fb0-47d4-a478-bb8a91fb3722 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:09:30.909502 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1f7dff6-6fb0-47d4-a478-bb8a91fb3722 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:09:30.915362 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc8122da-5fd5-435c-83b2-1188ef90282a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:09:30.916053 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc8122da-5fd5-435c-83b2-1188ef90282a admin admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:09:30.916689 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc8122da-5fd5-435c-83b2-1188ef90282a admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:09:30.916689 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc8122da-5fd5-435c-83b2-1188ef90282a admin admin] PATH_INFO: `/v3/domains` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:09:30.917431 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc8122da-5fd5-435c-83b2-1188ef90282a admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 02:09:30.917824 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc8122da-5fd5-435c-83b2-1188ef90282a admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:09:30.919769 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc8122da-5fd5-435c-83b2-1188ef90282a admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:09:30.942098 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1298/2595] 10.222.0.27 () {58 vars in 1153 bytes} [Tue Aug 30 02:09:30 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 49 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 02:09:30.951460 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cbe68ed-4315-4342-a567-db150a19607c None None] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:09:30.961296 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cbe68ed-4315-4342-a567-db150a19607c None None] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:09:30.967629 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3da821fb-d709-4eeb-84f4-52969d27715e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:09:30.968260 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3da821fb-d709-4eeb-84f4-52969d27715e admin admin] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:09:30.973535 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3da821fb-d709-4eeb-84f4-52969d27715e admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:09:30.973535 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3da821fb-d709-4eeb-84f4-52969d27715e admin admin] PATH_INFO: `/v3/projects` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:09:30.973535 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3da821fb-d709-4eeb-84f4-52969d27715e admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:09:30.973535 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3da821fb-d709-4eeb-84f4-52969d27715e admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:09:31.113325 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1298/2596] 10.222.0.27 () {60 vars in 1153 bytes} [Tue Aug 30 02:09:30 2022] POST /identity/v3/projects => generated 369 bytes in 160 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 30 02:09:31.115119 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc8122da-5fd5-435c-83b2-1188ef90282a admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:09:31.126730 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc8122da-5fd5-435c-83b2-1188ef90282a admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:09:31.192220 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd8f1d75-9ace-4bb6-aef6-4329597df3c5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:09:31.192844 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd8f1d75-9ace-4bb6-aef6-4329597df3c5 admin admin] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:09:31.193126 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd8f1d75-9ace-4bb6-aef6-4329597df3c5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:09:31.193400 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd8f1d75-9ace-4bb6-aef6-4329597df3c5 admin admin] PATH_INFO: `/v3/users` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:09:31.194346 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd8f1d75-9ace-4bb6-aef6-4329597df3c5 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:09:31.196044 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd8f1d75-9ace-4bb6-aef6-4329597df3c5 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:09:31.376730 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1299/2597] 10.222.0.27 () {60 vars in 1144 bytes} [Tue Aug 30 02:09:31 2022] POST /identity/v3/users => generated 285 bytes in 257 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 02:09:31.378181 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3da821fb-d709-4eeb-84f4-52969d27715e admin admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:09:31.389822 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3da821fb-d709-4eeb-84f4-52969d27715e admin admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:09:31.394787 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ddb1191-b066-40fa-a55b-0d72453fd0d7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:09:31.395402 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ddb1191-b066-40fa-a55b-0d72453fd0d7 admin admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:09:31.395683 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ddb1191-b066-40fa-a55b-0d72453fd0d7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:09:31.395950 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ddb1191-b066-40fa-a55b-0d72453fd0d7 admin admin] PATH_INFO: `/v3/roles` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:09:31.396606 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ddb1191-b066-40fa-a55b-0d72453fd0d7 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:09:31.398327 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ddb1191-b066-40fa-a55b-0d72453fd0d7 admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:09:31.410445 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1299/2598] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Aug 30 02:09:31 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 02:09:31.414325 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd8f1d75-9ace-4bb6-aef6-4329597df3c5 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:09:31.424560 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd8f1d75-9ace-4bb6-aef6-4329597df3c5 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:09:31.430169 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e963d9c1-1b2d-423d-a68e-e1c063db7442 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:09:31.430931 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e963d9c1-1b2d-423d-a68e-e1c063db7442 admin admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:09:31.431206 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e963d9c1-1b2d-423d-a68e-e1c063db7442 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:09:31.431500 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e963d9c1-1b2d-423d-a68e-e1c063db7442 admin admin] PATH_INFO: `/v3/roles` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:09:31.432222 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e963d9c1-1b2d-423d-a68e-e1c063db7442 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:09:31.433918 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e963d9c1-1b2d-423d-a68e-e1c063db7442 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:09:31.446231 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1300/2599] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Aug 30 02:09:31 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 02:09:31.450770 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ddb1191-b066-40fa-a55b-0d72453fd0d7 admin admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:09:31.461421 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ddb1191-b066-40fa-a55b-0d72453fd0d7 admin admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:09:31.474467 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d919882e-f59b-4b53-b44c-37d8915c1410 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:09:31.474467 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d919882e-f59b-4b53-b44c-37d8915c1410 admin admin] REQUEST_METHOD: `PUT` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:09:31.474467 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d919882e-f59b-4b53-b44c-37d8915c1410 admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:09:31.474467 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d919882e-f59b-4b53-b44c-37d8915c1410 admin admin] PATH_INFO: `/v3/projects/dbcd3b31535f46308cd1a15fd66538f0/users/ad16b4f1dfc94571bfcd4554ff588c86/roles/1fb81afcb9f245f3b989add94598a063` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:09:31.484747 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d919882e-f59b-4b53-b44c-37d8915c1410 admin admin] RBAC: Authorizing `identity:create_grant(project_id=dbcd3b31535f46308cd1a15fd66538f0, user_id=ad16b4f1dfc94571bfcd4554ff588c86, role_id=1fb81afcb9f245f3b989add94598a063)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:09:31.487381 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d919882e-f59b-4b53-b44c-37d8915c1410 admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:09:31.938940 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1300/2600] 10.222.0.27 () {60 vars in 1483 bytes} [Tue Aug 30 02:09:31 2022] PUT /identity/v3/projects/dbcd3b31535f46308cd1a15fd66538f0/users/ad16b4f1dfc94571bfcd4554ff588c86/roles/1fb81afcb9f245f3b989add94598a063 => generated 0 bytes in 489 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 02:09:31.944068 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb130647-4e36-4e98-b0ba-c33c8768af17 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:09:31.944539 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb130647-4e36-4e98-b0ba-c33c8768af17 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:09:31.944648 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb130647-4e36-4e98-b0ba-c33c8768af17 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:09:31.963587 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-fb130647-4e36-4e98-b0ba-c33c8768af17 None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:09:31.964774 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-fb130647-4e36-4e98-b0ba-c33c8768af17 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:09:32.035690 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1301/2601] 10.222.0.27 () {58 vars in 958 bytes} [Tue Aug 30 02:09:31 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 02:09:32.049393 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d919882e-f59b-4b53-b44c-37d8915c1410 admin admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:09:32.051201 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-d919882e-f59b-4b53-b44c-37d8915c1410 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:09:32.127331 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d919882e-f59b-4b53-b44c-37d8915c1410 admin admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:09:32.134944 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f8a9eb0-5995-410a-8817-c113dffa127f service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f3ea1580c56f4814a92bbb2acf542905', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4562f3951fbe465280073e60a11e44e2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:09:32.135620 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f8a9eb0-5995-410a-8817-c113dffa127f service neutron] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:09:32.136002 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f8a9eb0-5995-410a-8817-c113dffa127f service neutron] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:09:32.136385 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f8a9eb0-5995-410a-8817-c113dffa127f service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:09:32.149192 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f8a9eb0-5995-410a-8817-c113dffa127f service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:09:32.150909 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f8a9eb0-5995-410a-8817-c113dffa127f service neutron] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:09:32.167634 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1301/2602] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Aug 30 02:09:32 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 119 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 02:09:33.692178 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d622e5db-d166-45e3-83f6-3b1d2d9167b4 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:09:33.692744 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d622e5db-d166-45e3-83f6-3b1d2d9167b4 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:09:33.692922 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d622e5db-d166-45e3-83f6-3b1d2d9167b4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:09:33.730749 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-d622e5db-d166-45e3-83f6-3b1d2d9167b4 None None] MFA Rules not processed for user `ad16b4f1dfc94571bfcd4554ff588c86`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:09:33.732761 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-d622e5db-d166-45e3-83f6-3b1d2d9167b4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:09:33.773720 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:09:33.809642 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1302/2603] 10.222.0.27 () {58 vars in 958 bytes} [Tue Aug 30 02:09:33 2022] POST /identity/v3/auth/tokens => generated 3246 bytes in 119 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 02:09:33.821619 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f8a9eb0-5995-410a-8817-c113dffa127f service neutron] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:09:33.823084 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-9f8a9eb0-5995-410a-8817-c113dffa127f service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:09:33.884036 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f8a9eb0-5995-410a-8817-c113dffa127f service neutron] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:09:33.888206 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b834a0f-570d-4d09-ab49-3a1390e0ca91 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ab936b3767a8417aa80c8c8f1b84b667', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4562f3951fbe465280073e60a11e44e2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:09:33.888860 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b834a0f-570d-4d09-ab49-3a1390e0ca91 service cinder] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:09:33.889138 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b834a0f-570d-4d09-ab49-3a1390e0ca91 service cinder] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:09:33.889408 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b834a0f-570d-4d09-ab49-3a1390e0ca91 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:09:33.898632 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b834a0f-570d-4d09-ab49-3a1390e0ca91 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:09:33.899861 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b834a0f-570d-4d09-ab49-3a1390e0ca91 service cinder] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:09:33.912350 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1302/2604] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Aug 30 02:09:33 2022] GET /identity/v3/auth/tokens => generated 3246 bytes in 89 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 30 02:09:35.750244 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8e4b017-e866-4cbc-a24e-ca4a0ddd2c45 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:09:35.750722 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8e4b017-e866-4cbc-a24e-ca4a0ddd2c45 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:09:35.750818 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8e4b017-e866-4cbc-a24e-ca4a0ddd2c45 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:09:35.770657 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-d8e4b017-e866-4cbc-a24e-ca4a0ddd2c45 None None] MFA Rules not processed for user `ad16b4f1dfc94571bfcd4554ff588c86`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:09:35.771856 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-d8e4b017-e866-4cbc-a24e-ca4a0ddd2c45 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:09:35.778807 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1303/2605] 10.222.0.27 () {58 vars in 958 bytes} [Tue Aug 30 02:09:35 2022] POST /identity/v3/auth/tokens => generated 3246 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 02:09:38.348546 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84ec02ae-89d7-4f4c-b7e0-a1038e782643 None None] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:09:38.349232 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84ec02ae-89d7-4f4c-b7e0-a1038e782643 None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:09:38.349232 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84ec02ae-89d7-4f4c-b7e0-a1038e782643 None None] PATH_INFO: `/` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:09:38.350450 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1303/2606] 10.222.0.27 () {54 vars in 921 bytes} [Tue Aug 30 02:09:38 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 02:09:38.356627 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-059c2fa2-fa41-4984-bc6b-ffb032457c4d None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:09:38.357056 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-059c2fa2-fa41-4984-bc6b-ffb032457c4d None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:09:38.357242 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-059c2fa2-fa41-4984-bc6b-ffb032457c4d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:09:38.377947 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-059c2fa2-fa41-4984-bc6b-ffb032457c4d None None] MFA Rules not processed for user `d64f2e71361644589bec8a06e6eceed1`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:09:38.379146 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-059c2fa2-fa41-4984-bc6b-ffb032457c4d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:09:38.449164 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1304/2607] 10.222.0.27 () {58 vars in 1018 bytes} [Tue Aug 30 02:09:38 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 30 02:09:38.465739 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84ec02ae-89d7-4f4c-b7e0-a1038e782643 None None] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:09:38.467495 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-84ec02ae-89d7-4f4c-b7e0-a1038e782643 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:09:38.528085 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:09:38.534333 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84ec02ae-89d7-4f4c-b7e0-a1038e782643 None None] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:09:38.538102 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6979ee7d-35e5-4715-aec3-12599c183ea7 service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3caff62093d748f48596574836e12e42', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4562f3951fbe465280073e60a11e44e2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:09:38.538722 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6979ee7d-35e5-4715-aec3-12599c183ea7 service glance] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:09:38.539003 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6979ee7d-35e5-4715-aec3-12599c183ea7 service glance] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:09:38.539273 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6979ee7d-35e5-4715-aec3-12599c183ea7 service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:09:38.550389 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6979ee7d-35e5-4715-aec3-12599c183ea7 service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:09:38.551831 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6979ee7d-35e5-4715-aec3-12599c183ea7 service glance] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:09:38.563112 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1304/2608] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Aug 30 02:09:38 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 98 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 02:09:38.772146 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33174bca-2418-49fb-8559-9ed9f2425783 None None] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:09:38.772429 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33174bca-2418-49fb-8559-9ed9f2425783 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:09:38.772699 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33174bca-2418-49fb-8559-9ed9f2425783 None None] PATH_INFO: `/` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:09:38.774160 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1305/2609] 10.222.0.27 () {54 vars in 921 bytes} [Tue Aug 30 02:09:38 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 02:09:38.779097 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dd2fd36-b51c-4179-b6b9-46afeb4b8aae None None] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:09:38.779383 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dd2fd36-b51c-4179-b6b9-46afeb4b8aae None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:09:38.779785 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dd2fd36-b51c-4179-b6b9-46afeb4b8aae None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:09:38.800775 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.auth.core [None req-4dd2fd36-b51c-4179-b6b9-46afeb4b8aae None None] MFA Rules not processed for user `f3ea1580c56f4814a92bbb2acf542905`. Rule list: `[]` (Enabled: `True`). {{(pid=81578) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:09:38.802013 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-4dd2fd36-b51c-4179-b6b9-46afeb4b8aae None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:09:38.808527 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1305/2610] 10.222.0.27 () {58 vars in 1018 bytes} [Tue Aug 30 02:09:38 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 30 02:09:38.820446 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33174bca-2418-49fb-8559-9ed9f2425783 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:09:38.831592 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33174bca-2418-49fb-8559-9ed9f2425783 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:09:38.835131 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6902bd0e-f43c-40d8-b0e2-c2ce9490853c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f3ea1580c56f4814a92bbb2acf542905', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4562f3951fbe465280073e60a11e44e2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:09:38.835724 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6902bd0e-f43c-40d8-b0e2-c2ce9490853c service neutron] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:09:38.836005 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6902bd0e-f43c-40d8-b0e2-c2ce9490853c service neutron] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:09:38.836279 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6902bd0e-f43c-40d8-b0e2-c2ce9490853c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:09:38.844206 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6902bd0e-f43c-40d8-b0e2-c2ce9490853c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:09:38.845611 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6902bd0e-f43c-40d8-b0e2-c2ce9490853c service neutron] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:09:38.853872 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1306/2611] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Aug 30 02:09:38 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 34 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 02:09:38.935519 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad28dff6-6403-4b29-86ab-20970decca81 None None] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:09:38.935894 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad28dff6-6403-4b29-86ab-20970decca81 None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:09:38.936160 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad28dff6-6403-4b29-86ab-20970decca81 None None] PATH_INFO: `/` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:09:38.937724 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1306/2612] 10.222.0.27 () {54 vars in 921 bytes} [Tue Aug 30 02:09:38 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 02:09:38.942513 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7823d44e-014e-4d4c-8633-5947c8591502 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:09:38.942858 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7823d44e-014e-4d4c-8633-5947c8591502 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:09:38.947256 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7823d44e-014e-4d4c-8633-5947c8591502 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:09:38.964119 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-7823d44e-014e-4d4c-8633-5947c8591502 None None] MFA Rules not processed for user `d64f2e71361644589bec8a06e6eceed1`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:09:38.965351 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-7823d44e-014e-4d4c-8633-5947c8591502 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:09:38.975889 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1307/2613] 10.222.0.27 () {58 vars in 1018 bytes} [Tue Aug 30 02:09:38 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 02:09:38.987295 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad28dff6-6403-4b29-86ab-20970decca81 None None] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:09:38.998773 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad28dff6-6403-4b29-86ab-20970decca81 None None] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:09:39.002892 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d5f5d0b-bd30-42e4-84e2-6beeb40bd3d2 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f3ea1580c56f4814a92bbb2acf542905', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4562f3951fbe465280073e60a11e44e2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:09:39.003495 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d5f5d0b-bd30-42e4-84e2-6beeb40bd3d2 service neutron] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:09:39.003848 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d5f5d0b-bd30-42e4-84e2-6beeb40bd3d2 service neutron] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:09:39.004551 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d5f5d0b-bd30-42e4-84e2-6beeb40bd3d2 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:09:39.016006 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d5f5d0b-bd30-42e4-84e2-6beeb40bd3d2 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:09:39.017551 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d5f5d0b-bd30-42e4-84e2-6beeb40bd3d2 service neutron] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:09:39.028756 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1307/2614] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Aug 30 02:09:38 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 42 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 02:09:43.633651 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9c9369c-49e8-4617-ace0-e0b5aff0df61 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:09:43.634109 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9c9369c-49e8-4617-ace0-e0b5aff0df61 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:09:43.634203 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9c9369c-49e8-4617-ace0-e0b5aff0df61 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:09:43.652910 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-c9c9369c-49e8-4617-ace0-e0b5aff0df61 None None] MFA Rules not processed for user `121c374d49a64335ab74a431cfbc1320`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:09:43.654123 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-c9c9369c-49e8-4617-ace0-e0b5aff0df61 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:09:43.708483 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1308/2615] 10.222.0.27 () {58 vars in 1009 bytes} [Tue Aug 30 02:09:43 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 76 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 02:09:43.720211 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d5f5d0b-bd30-42e4-84e2-6beeb40bd3d2 service neutron] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:09:43.721642 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-5d5f5d0b-bd30-42e4-84e2-6beeb40bd3d2 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:09:43.789746 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d5f5d0b-bd30-42e4-84e2-6beeb40bd3d2 service neutron] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:09:43.794669 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33ec86a7-cb13-43ce-9359-6fe724e98913 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b4f0008fde424f22bc1f7307cf422774', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4562f3951fbe465280073e60a11e44e2', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:09:43.795264 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33ec86a7-cb13-43ce-9359-6fe724e98913 service swift] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:09:43.795534 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33ec86a7-cb13-43ce-9359-6fe724e98913 service swift] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:09:43.795800 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33ec86a7-cb13-43ce-9359-6fe724e98913 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:09:43.804193 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33ec86a7-cb13-43ce-9359-6fe724e98913 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:09:43.805717 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33ec86a7-cb13-43ce-9359-6fe724e98913 service swift] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:09:43.814117 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1308/2616] 10.222.0.27 () {60 vars in 1384 bytes} [Tue Aug 30 02:09:43 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 94 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Aug 30 02:09:53.701378 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=81 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:09:53.810393 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=46 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:10:02.584811 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4aaddf47-8689-4c19-98c3-0da0e6cee516 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:10:02.585376 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4aaddf47-8689-4c19-98c3-0da0e6cee516 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:10:02.585376 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4aaddf47-8689-4c19-98c3-0da0e6cee516 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:10:02.604995 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-4aaddf47-8689-4c19-98c3-0da0e6cee516 None None] MFA Rules not processed for user `ad16b4f1dfc94571bfcd4554ff588c86`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:10:02.606227 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-4aaddf47-8689-4c19-98c3-0da0e6cee516 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:10:02.615500 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1309/2617] 10.222.0.27 () {58 vars in 958 bytes} [Tue Aug 30 02:10:02 2022] POST /identity/v3/auth/tokens => generated 3246 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 02:10:09.690692 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-725f99cb-98a8-4dc7-be61-99c75ca2a3c4 None None] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:10:09.691270 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-725f99cb-98a8-4dc7-be61-99c75ca2a3c4 None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:10:09.691410 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-725f99cb-98a8-4dc7-be61-99c75ca2a3c4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:10:09.729413 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.auth.core [None req-725f99cb-98a8-4dc7-be61-99c75ca2a3c4 None None] MFA Rules not processed for user `121c374d49a64335ab74a431cfbc1320`. Rule list: `[]` (Enabled: `True`). {{(pid=81578) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:10:09.730643 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-725f99cb-98a8-4dc7-be61-99c75ca2a3c4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:10:09.753111 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1309/2618] 10.222.0.27 () {58 vars in 967 bytes} [Tue Aug 30 02:10:09 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 63 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 02:10:09.774124 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4aaddf47-8689-4c19-98c3-0da0e6cee516 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:10:09.788974 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4aaddf47-8689-4c19-98c3-0da0e6cee516 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:10:09.792057 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a23e479-dd63-40f4-94c6-8329539f356d service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b4f0008fde424f22bc1f7307cf422774', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4562f3951fbe465280073e60a11e44e2', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:10:09.792673 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a23e479-dd63-40f4-94c6-8329539f356d service swift] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:10:09.792953 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a23e479-dd63-40f4-94c6-8329539f356d service swift] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:10:09.793222 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a23e479-dd63-40f4-94c6-8329539f356d service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:10:09.803846 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a23e479-dd63-40f4-94c6-8329539f356d service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:10:09.805437 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a23e479-dd63-40f4-94c6-8329539f356d service swift] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:10:09.817691 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1310/2619] 10.222.0.27 () {60 vars in 1384 bytes} [Tue Aug 30 02:10:09 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 50 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Aug 30 02:10:15.117424 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-725f99cb-98a8-4dc7-be61-99c75ca2a3c4 None None] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:10:15.126430 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-725f99cb-98a8-4dc7-be61-99c75ca2a3c4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:10:15.149693 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-725f99cb-98a8-4dc7-be61-99c75ca2a3c4 None None] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:10:15.155481 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb45a49d-d947-4adc-b1ee-760d1a005e9d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:10:15.156123 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb45a49d-d947-4adc-b1ee-760d1a005e9d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:10:15.157029 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb45a49d-d947-4adc-b1ee-760d1a005e9d admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:10:15.157337 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb45a49d-d947-4adc-b1ee-760d1a005e9d admin admin] PATH_INFO: `/v3/users/c3557e6b6b2d46909db982ce76bbfdb8` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:10:15.158556 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb45a49d-d947-4adc-b1ee-760d1a005e9d admin admin] RBAC: Authorizing `identity:delete_user(user_id=c3557e6b6b2d46909db982ce76bbfdb8)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:10:15.162112 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb45a49d-d947-4adc-b1ee-760d1a005e9d admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:10:15.340551 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-cb45a49d-d947-4adc-b1ee-760d1a005e9d admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'c3557e6b6b2d46909db982ce76bbfdb8'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:10:15.341487 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-cb45a49d-d947-4adc-b1ee-760d1a005e9d admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'c3557e6b6b2d46909db982ce76bbfdb8'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:10:15.467124 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-cb45a49d-d947-4adc-b1ee-760d1a005e9d admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'c3557e6b6b2d46909db982ce76bbfdb8'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:10:15.467124 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-cb45a49d-d947-4adc-b1ee-760d1a005e9d admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'c3557e6b6b2d46909db982ce76bbfdb8'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:10:15.468433 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-cb45a49d-d947-4adc-b1ee-760d1a005e9d admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'c3557e6b6b2d46909db982ce76bbfdb8'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:10:15.489915 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1310/2620] 10.222.0.27 () {58 vars in 1224 bytes} [Tue Aug 30 02:10:15 2022] DELETE /identity/v3/users/c3557e6b6b2d46909db982ce76bbfdb8 => generated 0 bytes in 373 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 02:10:15.494561 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4b2416f-1c79-4c49-8361-a80b4418ae93 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:10:15.495113 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4b2416f-1c79-4c49-8361-a80b4418ae93 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:10:15.495338 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4b2416f-1c79-4c49-8361-a80b4418ae93 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:10:15.517582 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-a4b2416f-1c79-4c49-8361-a80b4418ae93 None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:10:15.518890 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-a4b2416f-1c79-4c49-8361-a80b4418ae93 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:10:15.580927 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1311/2621] 10.222.0.27 () {58 vars in 958 bytes} [Tue Aug 30 02:10:15 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 30 02:10:15.591370 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb45a49d-d947-4adc-b1ee-760d1a005e9d admin admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:10:15.592925 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-cb45a49d-d947-4adc-b1ee-760d1a005e9d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:10:15.657112 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb45a49d-d947-4adc-b1ee-760d1a005e9d admin admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:10:15.661071 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce46f77c-c5bf-4055-aec8-fe3c5f30b036 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f3ea1580c56f4814a92bbb2acf542905', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4562f3951fbe465280073e60a11e44e2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:10:15.661741 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce46f77c-c5bf-4055-aec8-fe3c5f30b036 service neutron] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:10:15.662083 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce46f77c-c5bf-4055-aec8-fe3c5f30b036 service neutron] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:10:15.662366 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce46f77c-c5bf-4055-aec8-fe3c5f30b036 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:10:15.673076 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce46f77c-c5bf-4055-aec8-fe3c5f30b036 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:10:15.674590 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce46f77c-c5bf-4055-aec8-fe3c5f30b036 service neutron] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:10:15.686767 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1311/2622] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Aug 30 02:10:15 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 96 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 02:10:16.180846 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4b2416f-1c79-4c49-8361-a80b4418ae93 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:10:16.182362 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-a4b2416f-1c79-4c49-8361-a80b4418ae93 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:10:16.195166 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4b2416f-1c79-4c49-8361-a80b4418ae93 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:10:16.200428 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6409f276-f513-4f7a-8f32-6b8143bc6d9b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:10:16.201068 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6409f276-f513-4f7a-8f32-6b8143bc6d9b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:10:16.201350 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6409f276-f513-4f7a-8f32-6b8143bc6d9b admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:10:16.201752 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6409f276-f513-4f7a-8f32-6b8143bc6d9b admin admin] PATH_INFO: `/v3/projects/7b8a54483ae04230ab5e0727c7eae0bc` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:10:16.204356 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6409f276-f513-4f7a-8f32-6b8143bc6d9b admin admin] RBAC: Authorizing `identity:delete_project(project_id=7b8a54483ae04230ab5e0727c7eae0bc)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:10:16.206345 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6409f276-f513-4f7a-8f32-6b8143bc6d9b admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:10:16.235535 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce46f77c-c5bf-4055-aec8-fe3c5f30b036 service neutron] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:10:16.237032 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-ce46f77c-c5bf-4055-aec8-fe3c5f30b036 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:10:16.304043 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:10:16.305749 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-6409f276-f513-4f7a-8f32-6b8143bc6d9b admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '7b8a54483ae04230ab5e0727c7eae0bc', 'request_id': 'req-6409f276-f513-4f7a-8f32-6b8143bc6d9b'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:10:16.311289 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-6409f276-f513-4f7a-8f32-6b8143bc6d9b admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '7b8a54483ae04230ab5e0727c7eae0bc', 'request_id': 'req-6409f276-f513-4f7a-8f32-6b8143bc6d9b'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:10:16.318000 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-6409f276-f513-4f7a-8f32-6b8143bc6d9b admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '7b8a54483ae04230ab5e0727c7eae0bc', 'request_id': 'req-6409f276-f513-4f7a-8f32-6b8143bc6d9b'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:10:16.337539 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce46f77c-c5bf-4055-aec8-fe3c5f30b036 service neutron] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:10:16.396613 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca7e2b62-cf96-4b34-bcef-de59fcded7eb service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '379536ff06fd4f719899e5dde732e63d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4562f3951fbe465280073e60a11e44e2', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:10:16.397215 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca7e2b62-cf96-4b34-bcef-de59fcded7eb service placement] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:10:16.397523 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca7e2b62-cf96-4b34-bcef-de59fcded7eb service placement] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:10:16.397796 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca7e2b62-cf96-4b34-bcef-de59fcded7eb service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:10:16.399520 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-ca7e2b62-cf96-4b34-bcef-de59fcded7eb service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:10:16.409909 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-6409f276-f513-4f7a-8f32-6b8143bc6d9b admin admin] The token cache is being invalidate because project 7b8a54483ae04230ab5e0727c7eae0bc was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81577) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 30 02:10:16.415234 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-6409f276-f513-4f7a-8f32-6b8143bc6d9b admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:10:16.415234 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-6409f276-f513-4f7a-8f32-6b8143bc6d9b admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:10:16.415494 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca7e2b62-cf96-4b34-bcef-de59fcded7eb service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:10:16.415629 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1312/2623] 10.222.0.27 () {58 vars in 1233 bytes} [Tue Aug 30 02:10:16 2022] DELETE /identity/v3/projects/7b8a54483ae04230ab5e0727c7eae0bc => generated 0 bytes in 232 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 02:10:16.415729 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca7e2b62-cf96-4b34-bcef-de59fcded7eb service placement] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:10:16.416426 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6409f276-f513-4f7a-8f32-6b8143bc6d9b admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:10:16.417786 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-6409f276-f513-4f7a-8f32-6b8143bc6d9b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:10:16.423120 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1312/2624] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Aug 30 02:10:16 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 188 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 30 02:10:16.484231 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6409f276-f513-4f7a-8f32-6b8143bc6d9b admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:10:16.489599 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97d7e6ad-86dd-456c-a307-4312a0ccc846 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:10:16.490232 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97d7e6ad-86dd-456c-a307-4312a0ccc846 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:10:16.490513 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97d7e6ad-86dd-456c-a307-4312a0ccc846 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:10:16.490783 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97d7e6ad-86dd-456c-a307-4312a0ccc846 admin admin] PATH_INFO: `/v3/users/5e5c6d54442d45b589f7a8daf0997aa3` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:10:16.491909 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97d7e6ad-86dd-456c-a307-4312a0ccc846 admin admin] RBAC: Authorizing `identity:delete_user(user_id=5e5c6d54442d45b589f7a8daf0997aa3)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:10:16.493857 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97d7e6ad-86dd-456c-a307-4312a0ccc846 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:10:16.498590 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:10:16.678998 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-97d7e6ad-86dd-456c-a307-4312a0ccc846 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '5e5c6d54442d45b589f7a8daf0997aa3'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:10:16.679679 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-97d7e6ad-86dd-456c-a307-4312a0ccc846 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '5e5c6d54442d45b589f7a8daf0997aa3'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:10:16.684829 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-97d7e6ad-86dd-456c-a307-4312a0ccc846 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '5e5c6d54442d45b589f7a8daf0997aa3'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:10:16.693568 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-97d7e6ad-86dd-456c-a307-4312a0ccc846 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '5e5c6d54442d45b589f7a8daf0997aa3'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:10:16.700456 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-97d7e6ad-86dd-456c-a307-4312a0ccc846 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '5e5c6d54442d45b589f7a8daf0997aa3'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:10:16.758378 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1313/2625] 10.222.0.27 () {58 vars in 1224 bytes} [Tue Aug 30 02:10:16 2022] DELETE /identity/v3/users/5e5c6d54442d45b589f7a8daf0997aa3 => generated 0 bytes in 342 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 02:10:17.265252 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca7e2b62-cf96-4b34-bcef-de59fcded7eb service placement] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:10:17.276256 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca7e2b62-cf96-4b34-bcef-de59fcded7eb service placement] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:10:17.335985 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ab9197f-5210-4777-ba80-8704e55fdaff admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:10:17.336628 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ab9197f-5210-4777-ba80-8704e55fdaff admin admin] REQUEST_METHOD: `DELETE` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:10:17.336909 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ab9197f-5210-4777-ba80-8704e55fdaff admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:10:17.337218 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ab9197f-5210-4777-ba80-8704e55fdaff admin admin] PATH_INFO: `/v3/projects/2fa7ea3ec2ee47c2929f6f7172bb31e9` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:10:17.338402 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ab9197f-5210-4777-ba80-8704e55fdaff admin admin] RBAC: Authorizing `identity:delete_project(project_id=2fa7ea3ec2ee47c2929f6f7172bb31e9)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:10:17.340299 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ab9197f-5210-4777-ba80-8704e55fdaff admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:10:17.454770 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-6ab9197f-5210-4777-ba80-8704e55fdaff admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '2fa7ea3ec2ee47c2929f6f7172bb31e9', 'request_id': 'req-6ab9197f-5210-4777-ba80-8704e55fdaff'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:10:17.461261 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-6ab9197f-5210-4777-ba80-8704e55fdaff admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '2fa7ea3ec2ee47c2929f6f7172bb31e9', 'request_id': 'req-6ab9197f-5210-4777-ba80-8704e55fdaff'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:10:17.468276 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-6ab9197f-5210-4777-ba80-8704e55fdaff admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '2fa7ea3ec2ee47c2929f6f7172bb31e9', 'request_id': 'req-6ab9197f-5210-4777-ba80-8704e55fdaff'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:10:17.473286 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:10:17.592862 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-6ab9197f-5210-4777-ba80-8704e55fdaff admin admin] The token cache is being invalidate because project 2fa7ea3ec2ee47c2929f6f7172bb31e9 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81578) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 30 02:10:17.593156 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-6ab9197f-5210-4777-ba80-8704e55fdaff admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:10:17.594404 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-6ab9197f-5210-4777-ba80-8704e55fdaff admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:10:17.596274 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1313/2626] 10.222.0.27 () {58 vars in 1233 bytes} [Tue Aug 30 02:10:17 2022] DELETE /identity/v3/projects/2fa7ea3ec2ee47c2929f6f7172bb31e9 => generated 0 bytes in 332 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 02:10:17.601125 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bee04bfb-d973-49e3-94aa-6dbf102cf3d0 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:10:17.601660 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bee04bfb-d973-49e3-94aa-6dbf102cf3d0 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:10:17.601770 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bee04bfb-d973-49e3-94aa-6dbf102cf3d0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:10:17.776650 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-bee04bfb-d973-49e3-94aa-6dbf102cf3d0 None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:10:17.777918 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-bee04bfb-d973-49e3-94aa-6dbf102cf3d0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:10:17.847182 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1314/2627] 10.222.0.27 () {58 vars in 958 bytes} [Tue Aug 30 02:10:17 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 243 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 02:10:17.851814 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82682d79-607a-4d52-9571-9a3549d4a72f None None] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:10:17.852104 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82682d79-607a-4d52-9571-9a3549d4a72f None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:10:17.852373 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82682d79-607a-4d52-9571-9a3549d4a72f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:10:17.880764 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.auth.core [None req-82682d79-607a-4d52-9571-9a3549d4a72f None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81578) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:10:17.882209 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-82682d79-607a-4d52-9571-9a3549d4a72f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:10:17.890451 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1314/2628] 10.222.0.27 () {58 vars in 958 bytes} [Tue Aug 30 02:10:17 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 40 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 02:10:17.895306 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bee04bfb-d973-49e3-94aa-6dbf102cf3d0 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:10:17.904960 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bee04bfb-d973-49e3-94aa-6dbf102cf3d0 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:10:17.911114 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73a80ef2-bb11-4c83-8b0e-cef09da6c11f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:10:17.911719 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73a80ef2-bb11-4c83-8b0e-cef09da6c11f admin admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:10:17.912025 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73a80ef2-bb11-4c83-8b0e-cef09da6c11f admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:10:17.912295 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73a80ef2-bb11-4c83-8b0e-cef09da6c11f admin admin] PATH_INFO: `/v3/domains` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:10:17.912974 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73a80ef2-bb11-4c83-8b0e-cef09da6c11f admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 02:10:17.913275 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73a80ef2-bb11-4c83-8b0e-cef09da6c11f admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:10:17.914944 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73a80ef2-bb11-4c83-8b0e-cef09da6c11f admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:10:17.926934 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1315/2629] 10.222.0.27 () {58 vars in 1153 bytes} [Tue Aug 30 02:10:17 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 02:10:17.931359 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82682d79-607a-4d52-9571-9a3549d4a72f None None] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:10:17.940827 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82682d79-607a-4d52-9571-9a3549d4a72f None None] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:10:17.945444 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdf0c5cc-5d20-4d2e-8b3b-85331c767a03 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:10:17.946066 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdf0c5cc-5d20-4d2e-8b3b-85331c767a03 admin admin] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:10:17.946344 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdf0c5cc-5d20-4d2e-8b3b-85331c767a03 admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:10:17.946610 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdf0c5cc-5d20-4d2e-8b3b-85331c767a03 admin admin] PATH_INFO: `/v3/projects` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:10:17.947228 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bdf0c5cc-5d20-4d2e-8b3b-85331c767a03 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:10:17.949078 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bdf0c5cc-5d20-4d2e-8b3b-85331c767a03 admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:10:18.035309 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1315/2630] 10.222.0.27 () {60 vars in 1153 bytes} [Tue Aug 30 02:10:17 2022] POST /identity/v3/projects => generated 399 bytes in 104 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 30 02:10:18.039431 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73a80ef2-bb11-4c83-8b0e-cef09da6c11f admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:10:18.049885 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73a80ef2-bb11-4c83-8b0e-cef09da6c11f admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:10:18.094718 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:10:18.108238 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26825371-39b6-4e85-a6f9-3e5d6022a847 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:10:18.108784 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26825371-39b6-4e85-a6f9-3e5d6022a847 admin admin] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:10:18.109063 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26825371-39b6-4e85-a6f9-3e5d6022a847 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:10:18.109342 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26825371-39b6-4e85-a6f9-3e5d6022a847 admin admin] PATH_INFO: `/v3/users` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:10:18.110122 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26825371-39b6-4e85-a6f9-3e5d6022a847 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:10:18.112174 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26825371-39b6-4e85-a6f9-3e5d6022a847 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:10:18.200435 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1316/2631] 10.222.0.27 () {60 vars in 1144 bytes} [Tue Aug 30 02:10:18 2022] POST /identity/v3/users => generated 300 bytes in 161 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 02:10:18.205792 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdf0c5cc-5d20-4d2e-8b3b-85331c767a03 admin admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:10:18.216914 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdf0c5cc-5d20-4d2e-8b3b-85331c767a03 admin admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:10:18.224274 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89303b78-2a4d-420a-b8fb-2c862bb17271 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:10:18.225031 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89303b78-2a4d-420a-b8fb-2c862bb17271 admin admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:10:18.231838 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89303b78-2a4d-420a-b8fb-2c862bb17271 admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:10:18.231838 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89303b78-2a4d-420a-b8fb-2c862bb17271 admin admin] PATH_INFO: `/v3/roles` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:10:18.231838 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89303b78-2a4d-420a-b8fb-2c862bb17271 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:10:18.231838 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89303b78-2a4d-420a-b8fb-2c862bb17271 admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:10:18.241923 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1316/2632] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Aug 30 02:10:18 2022] GET /identity/v3/roles => generated 1420 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 30 02:10:18.245735 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26825371-39b6-4e85-a6f9-3e5d6022a847 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:10:18.256921 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26825371-39b6-4e85-a6f9-3e5d6022a847 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:10:18.262038 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b26b5eee-b2b9-4622-8901-fc1b96f5002e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:10:18.262551 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b26b5eee-b2b9-4622-8901-fc1b96f5002e admin admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:10:18.262845 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b26b5eee-b2b9-4622-8901-fc1b96f5002e admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:10:18.263102 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b26b5eee-b2b9-4622-8901-fc1b96f5002e admin admin] PATH_INFO: `/v3/roles` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:10:18.263739 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b26b5eee-b2b9-4622-8901-fc1b96f5002e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:10:18.265348 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b26b5eee-b2b9-4622-8901-fc1b96f5002e admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:10:18.276773 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1317/2633] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Aug 30 02:10:18 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 02:10:18.281141 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89303b78-2a4d-420a-b8fb-2c862bb17271 admin admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:10:18.292461 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89303b78-2a4d-420a-b8fb-2c862bb17271 admin admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:10:18.297194 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10914b35-f652-4ab0-913e-42151c015e79 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:10:18.298043 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10914b35-f652-4ab0-913e-42151c015e79 admin admin] REQUEST_METHOD: `PUT` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:10:18.298319 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10914b35-f652-4ab0-913e-42151c015e79 admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:10:18.298593 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10914b35-f652-4ab0-913e-42151c015e79 admin admin] PATH_INFO: `/v3/projects/3e370a58098c48fcb56c8fb41e47b462/users/19761e9953454cd6a63170956c4bbdab/roles/1fb81afcb9f245f3b989add94598a063` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:10:18.315552 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10914b35-f652-4ab0-913e-42151c015e79 admin admin] RBAC: Authorizing `identity:create_grant(project_id=3e370a58098c48fcb56c8fb41e47b462, user_id=19761e9953454cd6a63170956c4bbdab, role_id=1fb81afcb9f245f3b989add94598a063)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:10:18.318331 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10914b35-f652-4ab0-913e-42151c015e79 admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:10:18.401273 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1317/2634] 10.222.0.27 () {60 vars in 1483 bytes} [Tue Aug 30 02:10:18 2022] PUT /identity/v3/projects/3e370a58098c48fcb56c8fb41e47b462/users/19761e9953454cd6a63170956c4bbdab/roles/1fb81afcb9f245f3b989add94598a063 => generated 0 bytes in 119 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 30 02:10:18.411367 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2496f66e-3a7e-479d-b2a9-346e1509ac71 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:10:18.411658 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2496f66e-3a7e-479d-b2a9-346e1509ac71 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:10:18.411936 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2496f66e-3a7e-479d-b2a9-346e1509ac71 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:10:18.440056 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-2496f66e-3a7e-479d-b2a9-346e1509ac71 None None] MFA Rules not processed for user `19761e9953454cd6a63170956c4bbdab`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:10:18.441275 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-2496f66e-3a7e-479d-b2a9-346e1509ac71 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:10:18.504589 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1318/2635] 10.222.0.27 () {58 vars in 958 bytes} [Tue Aug 30 02:10:18 2022] POST /identity/v3/auth/tokens => generated 3276 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 02:10:18.509114 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10914b35-f652-4ab0-913e-42151c015e79 admin admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:10:18.579788 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10914b35-f652-4ab0-913e-42151c015e79 admin admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:10:18.584883 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17b8c5c2-83f3-409b-8eea-828d14d18397 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:10:18.585511 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17b8c5c2-83f3-409b-8eea-828d14d18397 admin admin] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:10:18.585793 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17b8c5c2-83f3-409b-8eea-828d14d18397 admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:10:18.586102 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17b8c5c2-83f3-409b-8eea-828d14d18397 admin admin] PATH_INFO: `/v3/projects` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:10:18.586773 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17b8c5c2-83f3-409b-8eea-828d14d18397 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:10:18.588730 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17b8c5c2-83f3-409b-8eea-828d14d18397 admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:10:18.692751 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1318/2636] 10.222.0.27 () {60 vars in 1153 bytes} [Tue Aug 30 02:10:18 2022] POST /identity/v3/projects => generated 401 bytes in 184 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 02:10:18.696944 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2496f66e-3a7e-479d-b2a9-346e1509ac71 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:10:18.708591 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2496f66e-3a7e-479d-b2a9-346e1509ac71 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:10:18.765107 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-425ea95c-1ff9-44e1-a943-132ef15502a8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:10:18.765739 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-425ea95c-1ff9-44e1-a943-132ef15502a8 admin admin] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:10:18.766022 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-425ea95c-1ff9-44e1-a943-132ef15502a8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:10:18.766308 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-425ea95c-1ff9-44e1-a943-132ef15502a8 admin admin] PATH_INFO: `/v3/users` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:10:18.766937 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-425ea95c-1ff9-44e1-a943-132ef15502a8 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:10:18.768846 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-425ea95c-1ff9-44e1-a943-132ef15502a8 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:10:18.873158 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1319/2637] 10.222.0.27 () {60 vars in 1144 bytes} [Tue Aug 30 02:10:18 2022] POST /identity/v3/users => generated 307 bytes in 174 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 02:10:18.874872 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17b8c5c2-83f3-409b-8eea-828d14d18397 admin admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:10:18.896592 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17b8c5c2-83f3-409b-8eea-828d14d18397 admin admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:10:18.908326 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90b38213-7f52-4bd5-9579-4656b8819a3e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:10:18.908986 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90b38213-7f52-4bd5-9579-4656b8819a3e admin admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:10:18.909307 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90b38213-7f52-4bd5-9579-4656b8819a3e admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:10:18.909661 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90b38213-7f52-4bd5-9579-4656b8819a3e admin admin] PATH_INFO: `/v3/roles` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:10:18.911800 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90b38213-7f52-4bd5-9579-4656b8819a3e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:10:18.913069 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90b38213-7f52-4bd5-9579-4656b8819a3e admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:10:18.926634 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1319/2638] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Aug 30 02:10:18 2022] GET /identity/v3/roles => generated 1420 bytes in 52 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 02:10:18.930867 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-425ea95c-1ff9-44e1-a943-132ef15502a8 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:10:18.941943 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-425ea95c-1ff9-44e1-a943-132ef15502a8 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:10:18.949046 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-420cfdf5-cf8c-4fc2-96d7-790ca374f9e1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:10:18.949046 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-420cfdf5-cf8c-4fc2-96d7-790ca374f9e1 admin admin] REQUEST_METHOD: `PUT` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:10:18.949046 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-420cfdf5-cf8c-4fc2-96d7-790ca374f9e1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:10:18.949046 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-420cfdf5-cf8c-4fc2-96d7-790ca374f9e1 admin admin] PATH_INFO: `/v3/projects/f66db09a94e440e59fb67c3fadecbe4e/users/2eddc0237b1d415ebc7c1bcc388d3e96/roles/ead5fac86a0747ea8823a82efcea8ed7` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:10:18.962988 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-420cfdf5-cf8c-4fc2-96d7-790ca374f9e1 admin admin] RBAC: Authorizing `identity:create_grant(project_id=f66db09a94e440e59fb67c3fadecbe4e, user_id=2eddc0237b1d415ebc7c1bcc388d3e96, role_id=ead5fac86a0747ea8823a82efcea8ed7)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:10:18.965671 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-420cfdf5-cf8c-4fc2-96d7-790ca374f9e1 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:10:19.087277 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1320/2639] 10.222.0.27 () {60 vars in 1483 bytes} [Tue Aug 30 02:10:18 2022] PUT /identity/v3/projects/f66db09a94e440e59fb67c3fadecbe4e/users/2eddc0237b1d415ebc7c1bcc388d3e96/roles/ead5fac86a0747ea8823a82efcea8ed7 => generated 0 bytes in 157 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 02:10:19.091275 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90b38213-7f52-4bd5-9579-4656b8819a3e admin admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:10:19.101347 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90b38213-7f52-4bd5-9579-4656b8819a3e admin admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:10:19.158662 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5709b6a7-81a4-4ad5-a7a7-c20adcd47de0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:10:19.159262 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5709b6a7-81a4-4ad5-a7a7-c20adcd47de0 admin admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:10:19.159594 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5709b6a7-81a4-4ad5-a7a7-c20adcd47de0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:10:19.159866 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5709b6a7-81a4-4ad5-a7a7-c20adcd47de0 admin admin] PATH_INFO: `/v3/roles` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:10:19.160498 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5709b6a7-81a4-4ad5-a7a7-c20adcd47de0 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:10:19.162263 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5709b6a7-81a4-4ad5-a7a7-c20adcd47de0 admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:10:19.172022 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1320/2640] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Aug 30 02:10:19 2022] GET /identity/v3/roles => generated 1420 bytes in 81 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 02:10:19.176442 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-420cfdf5-cf8c-4fc2-96d7-790ca374f9e1 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:10:19.186931 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-420cfdf5-cf8c-4fc2-96d7-790ca374f9e1 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:10:19.192129 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a55e6b2-b76c-4122-b86f-459412faf330 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:10:19.192779 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a55e6b2-b76c-4122-b86f-459412faf330 admin admin] REQUEST_METHOD: `PUT` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:10:19.193058 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a55e6b2-b76c-4122-b86f-459412faf330 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:10:19.193449 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a55e6b2-b76c-4122-b86f-459412faf330 admin admin] PATH_INFO: `/v3/projects/f66db09a94e440e59fb67c3fadecbe4e/users/2eddc0237b1d415ebc7c1bcc388d3e96/roles/ead5fac86a0747ea8823a82efcea8ed7` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:10:19.195423 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a55e6b2-b76c-4122-b86f-459412faf330 admin admin] RBAC: Authorizing `identity:create_grant(project_id=f66db09a94e440e59fb67c3fadecbe4e, user_id=2eddc0237b1d415ebc7c1bcc388d3e96, role_id=ead5fac86a0747ea8823a82efcea8ed7)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:10:19.198081 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a55e6b2-b76c-4122-b86f-459412faf330 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:10:19.215478 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1321/2641] 10.222.0.27 () {60 vars in 1483 bytes} [Tue Aug 30 02:10:19 2022] PUT /identity/v3/projects/f66db09a94e440e59fb67c3fadecbe4e/users/2eddc0237b1d415ebc7c1bcc388d3e96/roles/ead5fac86a0747ea8823a82efcea8ed7 => generated 0 bytes in 33 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 02:10:19.219577 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-255ec850-1b98-4cce-86ef-a13b42fe9715 None None] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:10:19.219868 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-255ec850-1b98-4cce-86ef-a13b42fe9715 None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:10:19.220142 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-255ec850-1b98-4cce-86ef-a13b42fe9715 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:10:19.227299 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:10:19.248505 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.auth.core [None req-255ec850-1b98-4cce-86ef-a13b42fe9715 None None] MFA Rules not processed for user `2eddc0237b1d415ebc7c1bcc388d3e96`. Rule list: `[]` (Enabled: `True`). {{(pid=81578) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:10:19.250286 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-255ec850-1b98-4cce-86ef-a13b42fe9715 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:10:19.321185 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1321/2642] 10.222.0.27 () {58 vars in 958 bytes} [Tue Aug 30 02:10:19 2022] POST /identity/v3/auth/tokens => generated 3345 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 02:10:19.332745 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a55e6b2-b76c-4122-b86f-459412faf330 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:10:19.334475 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-5a55e6b2-b76c-4122-b86f-459412faf330 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:10:19.345395 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=92,keystone:INSERT=8 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:10:19.398219 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a55e6b2-b76c-4122-b86f-459412faf330 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:10:19.402007 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b7c9ad1-7b8e-4440-92ce-b8a6290f33f0 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ab936b3767a8417aa80c8c8f1b84b667', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4562f3951fbe465280073e60a11e44e2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:10:19.402648 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b7c9ad1-7b8e-4440-92ce-b8a6290f33f0 service cinder] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:10:19.402934 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b7c9ad1-7b8e-4440-92ce-b8a6290f33f0 service cinder] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:10:19.403202 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b7c9ad1-7b8e-4440-92ce-b8a6290f33f0 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:10:19.412965 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b7c9ad1-7b8e-4440-92ce-b8a6290f33f0 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:10:19.414421 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b7c9ad1-7b8e-4440-92ce-b8a6290f33f0 service cinder] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:10:19.424217 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1322/2643] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Aug 30 02:10:19 2022] GET /identity/v3/auth/tokens => generated 3345 bytes in 92 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 02:10:19.631649 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-255ec850-1b98-4cce-86ef-a13b42fe9715 None None] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:10:19.642772 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-255ec850-1b98-4cce-86ef-a13b42fe9715 None None] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:10:19.646405 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-630fa30c-a4e2-4430-b4cf-a0b3030812d0 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ab936b3767a8417aa80c8c8f1b84b667', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4562f3951fbe465280073e60a11e44e2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:10:19.647025 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-630fa30c-a4e2-4430-b4cf-a0b3030812d0 service cinder] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:10:19.647302 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-630fa30c-a4e2-4430-b4cf-a0b3030812d0 service cinder] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:10:19.647572 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-630fa30c-a4e2-4430-b4cf-a0b3030812d0 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:10:19.706519 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-630fa30c-a4e2-4430-b4cf-a0b3030812d0 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:10:19.707995 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-630fa30c-a4e2-4430-b4cf-a0b3030812d0 service cinder] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:10:19.717226 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1322/2644] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Aug 30 02:10:19 2022] GET /identity/v3/auth/tokens => generated 3276 bytes in 86 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 02:10:24.824832 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b7c9ad1-7b8e-4440-92ce-b8a6290f33f0 service cinder] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:10:24.886736 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b7c9ad1-7b8e-4440-92ce-b8a6290f33f0 service cinder] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:10:24.893397 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60ae5622-1faf-485f-85bb-659c37efe09d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:10:24.894558 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60ae5622-1faf-485f-85bb-659c37efe09d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:10:24.894558 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60ae5622-1faf-485f-85bb-659c37efe09d admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:10:24.894741 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60ae5622-1faf-485f-85bb-659c37efe09d admin admin] PATH_INFO: `/v3/users/19761e9953454cd6a63170956c4bbdab` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:10:24.895707 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60ae5622-1faf-485f-85bb-659c37efe09d admin admin] RBAC: Authorizing `identity:delete_user(user_id=19761e9953454cd6a63170956c4bbdab)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:10:24.897678 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60ae5622-1faf-485f-85bb-659c37efe09d admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:10:25.056344 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-60ae5622-1faf-485f-85bb-659c37efe09d admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '19761e9953454cd6a63170956c4bbdab'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:10:25.056568 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-60ae5622-1faf-485f-85bb-659c37efe09d admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '19761e9953454cd6a63170956c4bbdab'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:10:25.057329 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-60ae5622-1faf-485f-85bb-659c37efe09d admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '19761e9953454cd6a63170956c4bbdab'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:10:25.070556 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-60ae5622-1faf-485f-85bb-659c37efe09d admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '19761e9953454cd6a63170956c4bbdab'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:10:25.077557 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-60ae5622-1faf-485f-85bb-659c37efe09d admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '19761e9953454cd6a63170956c4bbdab'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:10:25.155273 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1323/2645] 10.222.0.27 () {58 vars in 1224 bytes} [Tue Aug 30 02:10:24 2022] DELETE /identity/v3/users/19761e9953454cd6a63170956c4bbdab => generated 0 bytes in 331 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 02:10:25.160534 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa1cf5c8-dbda-4d7c-aaad-66f3c220c74d None None] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:10:25.161010 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa1cf5c8-dbda-4d7c-aaad-66f3c220c74d None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:10:25.161106 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa1cf5c8-dbda-4d7c-aaad-66f3c220c74d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:10:25.180791 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.auth.core [None req-aa1cf5c8-dbda-4d7c-aaad-66f3c220c74d None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81578) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:10:25.182044 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-aa1cf5c8-dbda-4d7c-aaad-66f3c220c74d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:10:25.245174 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1323/2646] 10.222.0.27 () {58 vars in 958 bytes} [Tue Aug 30 02:10:25 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 02:10:25.254879 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60ae5622-1faf-485f-85bb-659c37efe09d admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:10:25.256489 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-60ae5622-1faf-485f-85bb-659c37efe09d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:10:25.288469 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:10:25.313817 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60ae5622-1faf-485f-85bb-659c37efe09d admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:10:25.317358 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca0f4e9f-059f-4c10-9745-f0a8a62d7cd2 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f3ea1580c56f4814a92bbb2acf542905', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4562f3951fbe465280073e60a11e44e2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:10:25.321678 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca0f4e9f-059f-4c10-9745-f0a8a62d7cd2 service neutron] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:10:25.321678 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca0f4e9f-059f-4c10-9745-f0a8a62d7cd2 service neutron] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:10:25.321678 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca0f4e9f-059f-4c10-9745-f0a8a62d7cd2 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:10:25.327457 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca0f4e9f-059f-4c10-9745-f0a8a62d7cd2 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:10:25.328933 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca0f4e9f-059f-4c10-9745-f0a8a62d7cd2 service neutron] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:10:25.340103 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1324/2647] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Aug 30 02:10:25 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 02:10:25.757931 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa1cf5c8-dbda-4d7c-aaad-66f3c220c74d None None] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:10:25.759318 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-aa1cf5c8-dbda-4d7c-aaad-66f3c220c74d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:10:25.773893 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa1cf5c8-dbda-4d7c-aaad-66f3c220c74d None None] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:10:25.778590 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e575d091-6112-4577-8e6f-36ea92876fc0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:10:25.779223 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e575d091-6112-4577-8e6f-36ea92876fc0 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:10:25.780178 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e575d091-6112-4577-8e6f-36ea92876fc0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:10:25.780178 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e575d091-6112-4577-8e6f-36ea92876fc0 admin admin] PATH_INFO: `/v3/projects/3e370a58098c48fcb56c8fb41e47b462` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:10:25.781328 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e575d091-6112-4577-8e6f-36ea92876fc0 admin admin] RBAC: Authorizing `identity:delete_project(project_id=3e370a58098c48fcb56c8fb41e47b462)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:10:25.783881 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e575d091-6112-4577-8e6f-36ea92876fc0 admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:10:25.807623 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:10:25.902428 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-e575d091-6112-4577-8e6f-36ea92876fc0 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '3e370a58098c48fcb56c8fb41e47b462', 'request_id': 'req-e575d091-6112-4577-8e6f-36ea92876fc0'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:10:25.908050 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-e575d091-6112-4577-8e6f-36ea92876fc0 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '3e370a58098c48fcb56c8fb41e47b462', 'request_id': 'req-e575d091-6112-4577-8e6f-36ea92876fc0'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:10:25.914298 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-e575d091-6112-4577-8e6f-36ea92876fc0 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '3e370a58098c48fcb56c8fb41e47b462', 'request_id': 'req-e575d091-6112-4577-8e6f-36ea92876fc0'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:10:25.994918 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-e575d091-6112-4577-8e6f-36ea92876fc0 admin admin] The token cache is being invalidate because project 3e370a58098c48fcb56c8fb41e47b462 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81578) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 30 02:10:25.995217 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-e575d091-6112-4577-8e6f-36ea92876fc0 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:10:25.995810 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-e575d091-6112-4577-8e6f-36ea92876fc0 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:10:25.997298 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1324/2648] 10.222.0.27 () {58 vars in 1233 bytes} [Tue Aug 30 02:10:25 2022] DELETE /identity/v3/projects/3e370a58098c48fcb56c8fb41e47b462 => generated 0 bytes in 240 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 02:10:26.001868 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca0f4e9f-059f-4c10-9745-f0a8a62d7cd2 service neutron] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:10:26.011220 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca0f4e9f-059f-4c10-9745-f0a8a62d7cd2 service neutron] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:10:26.067685 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4584b77-fba0-4827-83b4-53b8fa0cae81 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:10:26.068238 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4584b77-fba0-4827-83b4-53b8fa0cae81 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:10:26.068519 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4584b77-fba0-4827-83b4-53b8fa0cae81 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:10:26.068798 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4584b77-fba0-4827-83b4-53b8fa0cae81 admin admin] PATH_INFO: `/v3/users/2eddc0237b1d415ebc7c1bcc388d3e96` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:10:26.069936 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4584b77-fba0-4827-83b4-53b8fa0cae81 admin admin] RBAC: Authorizing `identity:delete_user(user_id=2eddc0237b1d415ebc7c1bcc388d3e96)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:10:26.071844 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4584b77-fba0-4827-83b4-53b8fa0cae81 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:10:26.225428 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-a4584b77-fba0-4827-83b4-53b8fa0cae81 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '2eddc0237b1d415ebc7c1bcc388d3e96'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:10:26.226097 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-a4584b77-fba0-4827-83b4-53b8fa0cae81 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '2eddc0237b1d415ebc7c1bcc388d3e96'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:10:26.226701 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-a4584b77-fba0-4827-83b4-53b8fa0cae81 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '2eddc0237b1d415ebc7c1bcc388d3e96'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:10:26.239063 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-a4584b77-fba0-4827-83b4-53b8fa0cae81 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '2eddc0237b1d415ebc7c1bcc388d3e96'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:10:26.245744 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-a4584b77-fba0-4827-83b4-53b8fa0cae81 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '2eddc0237b1d415ebc7c1bcc388d3e96'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:10:26.309060 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1325/2649] 10.222.0.27 () {58 vars in 1224 bytes} [Tue Aug 30 02:10:26 2022] DELETE /identity/v3/users/2eddc0237b1d415ebc7c1bcc388d3e96 => generated 0 bytes in 308 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 02:10:26.699436 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e575d091-6112-4577-8e6f-36ea92876fc0 admin admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:10:26.700915 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-e575d091-6112-4577-8e6f-36ea92876fc0 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:10:26.718961 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e575d091-6112-4577-8e6f-36ea92876fc0 admin admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:10:26.781245 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f76ee576-7e30-4cf2-9b9e-8d4cda4f09f1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:10:26.782397 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f76ee576-7e30-4cf2-9b9e-8d4cda4f09f1 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:10:26.782711 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f76ee576-7e30-4cf2-9b9e-8d4cda4f09f1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:10:26.783049 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f76ee576-7e30-4cf2-9b9e-8d4cda4f09f1 admin admin] PATH_INFO: `/v3/projects/f66db09a94e440e59fb67c3fadecbe4e` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:10:26.784157 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f76ee576-7e30-4cf2-9b9e-8d4cda4f09f1 admin admin] RBAC: Authorizing `identity:delete_project(project_id=f66db09a94e440e59fb67c3fadecbe4e)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:10:26.786090 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f76ee576-7e30-4cf2-9b9e-8d4cda4f09f1 admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:10:26.887050 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-f76ee576-7e30-4cf2-9b9e-8d4cda4f09f1 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'f66db09a94e440e59fb67c3fadecbe4e', 'request_id': 'req-f76ee576-7e30-4cf2-9b9e-8d4cda4f09f1'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:10:26.893216 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-f76ee576-7e30-4cf2-9b9e-8d4cda4f09f1 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'f66db09a94e440e59fb67c3fadecbe4e', 'request_id': 'req-f76ee576-7e30-4cf2-9b9e-8d4cda4f09f1'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:10:26.899419 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-f76ee576-7e30-4cf2-9b9e-8d4cda4f09f1 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'f66db09a94e440e59fb67c3fadecbe4e', 'request_id': 'req-f76ee576-7e30-4cf2-9b9e-8d4cda4f09f1'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:10:26.965854 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-f76ee576-7e30-4cf2-9b9e-8d4cda4f09f1 admin admin] The token cache is being invalidate because project f66db09a94e440e59fb67c3fadecbe4e was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81578) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 30 02:10:26.966124 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-f76ee576-7e30-4cf2-9b9e-8d4cda4f09f1 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:10:26.966733 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-f76ee576-7e30-4cf2-9b9e-8d4cda4f09f1 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:10:26.968221 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1325/2650] 10.222.0.27 () {58 vars in 1233 bytes} [Tue Aug 30 02:10:26 2022] DELETE /identity/v3/projects/f66db09a94e440e59fb67c3fadecbe4e => generated 0 bytes in 269 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 02:10:26.974222 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c999249-fc49-49a6-9857-ba682510e163 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:10:26.974510 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c999249-fc49-49a6-9857-ba682510e163 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:10:26.974782 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c999249-fc49-49a6-9857-ba682510e163 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:10:26.994692 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-3c999249-fc49-49a6-9857-ba682510e163 None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:10:26.996019 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-3c999249-fc49-49a6-9857-ba682510e163 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:10:27.042395 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:10:27.056578 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1326/2651] 10.222.0.27 () {58 vars in 958 bytes} [Tue Aug 30 02:10:26 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 02:10:27.064354 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3011c706-019d-4079-a6f5-76bfe1b1799d None None] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:10:27.064631 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3011c706-019d-4079-a6f5-76bfe1b1799d None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:10:27.064903 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3011c706-019d-4079-a6f5-76bfe1b1799d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:10:27.085623 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.auth.core [None req-3011c706-019d-4079-a6f5-76bfe1b1799d None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81578) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:10:27.086848 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-3011c706-019d-4079-a6f5-76bfe1b1799d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:10:27.095451 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1326/2652] 10.222.0.27 () {58 vars in 958 bytes} [Tue Aug 30 02:10:27 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 02:10:27.099614 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c999249-fc49-49a6-9857-ba682510e163 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:10:27.110169 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c999249-fc49-49a6-9857-ba682510e163 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:10:27.114927 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-baf2d644-5ff0-46d5-a1f0-0847531b8cc3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:10:27.115529 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-baf2d644-5ff0-46d5-a1f0-0847531b8cc3 admin admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:10:27.115871 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-baf2d644-5ff0-46d5-a1f0-0847531b8cc3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:10:27.116267 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-baf2d644-5ff0-46d5-a1f0-0847531b8cc3 admin admin] PATH_INFO: `/v3/domains` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:10:27.116921 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-baf2d644-5ff0-46d5-a1f0-0847531b8cc3 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 02:10:27.117223 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-baf2d644-5ff0-46d5-a1f0-0847531b8cc3 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:10:27.118928 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-baf2d644-5ff0-46d5-a1f0-0847531b8cc3 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:10:27.131101 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1327/2653] 10.222.0.27 () {58 vars in 1153 bytes} [Tue Aug 30 02:10:27 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 02:10:27.135619 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3011c706-019d-4079-a6f5-76bfe1b1799d None None] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:10:27.145632 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3011c706-019d-4079-a6f5-76bfe1b1799d None None] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:10:27.150343 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76149e14-fb65-4199-bdd0-2422fd3b0406 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:10:27.150942 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76149e14-fb65-4199-bdd0-2422fd3b0406 admin admin] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:10:27.151218 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76149e14-fb65-4199-bdd0-2422fd3b0406 admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:10:27.151485 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76149e14-fb65-4199-bdd0-2422fd3b0406 admin admin] PATH_INFO: `/v3/projects` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:10:27.152121 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76149e14-fb65-4199-bdd0-2422fd3b0406 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:10:27.154023 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76149e14-fb65-4199-bdd0-2422fd3b0406 admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:10:27.243495 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1327/2654] 10.222.0.27 () {60 vars in 1153 bytes} [Tue Aug 30 02:10:27 2022] POST /identity/v3/projects => generated 381 bytes in 108 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 02:10:27.247484 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-baf2d644-5ff0-46d5-a1f0-0847531b8cc3 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:10:27.257347 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-baf2d644-5ff0-46d5-a1f0-0847531b8cc3 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:10:27.312043 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad510bf2-518f-4eb7-8694-a8c73f415564 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:10:27.312640 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad510bf2-518f-4eb7-8694-a8c73f415564 admin admin] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:10:27.312916 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad510bf2-518f-4eb7-8694-a8c73f415564 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:10:27.313721 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad510bf2-518f-4eb7-8694-a8c73f415564 admin admin] PATH_INFO: `/v3/users` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:10:27.314445 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad510bf2-518f-4eb7-8694-a8c73f415564 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:10:27.316327 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad510bf2-518f-4eb7-8694-a8c73f415564 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:10:27.461801 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1328/2655] 10.222.0.27 () {60 vars in 1144 bytes} [Tue Aug 30 02:10:27 2022] POST /identity/v3/users => generated 291 bytes in 215 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 02:10:27.465875 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76149e14-fb65-4199-bdd0-2422fd3b0406 admin admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:10:27.476306 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76149e14-fb65-4199-bdd0-2422fd3b0406 admin admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:10:27.481252 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7b2467e-781b-488f-96c8-6a1eea746171 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:10:27.481904 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7b2467e-781b-488f-96c8-6a1eea746171 admin admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:10:27.482186 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7b2467e-781b-488f-96c8-6a1eea746171 admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:10:27.482455 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7b2467e-781b-488f-96c8-6a1eea746171 admin admin] PATH_INFO: `/v3/roles` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:10:27.483095 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7b2467e-781b-488f-96c8-6a1eea746171 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:10:27.484662 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7b2467e-781b-488f-96c8-6a1eea746171 admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:10:27.489930 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:DELETE=14,keystone:SELECT=83,keystone:INSERT=3 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:10:27.494762 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1328/2656] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Aug 30 02:10:27 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 02:10:27.498842 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad510bf2-518f-4eb7-8694-a8c73f415564 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:10:27.509564 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad510bf2-518f-4eb7-8694-a8c73f415564 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:10:27.514218 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73cd3c41-a529-44f4-9d2e-470d3cba8011 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:10:27.514801 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73cd3c41-a529-44f4-9d2e-470d3cba8011 admin admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:10:27.515077 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73cd3c41-a529-44f4-9d2e-470d3cba8011 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:10:27.515347 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73cd3c41-a529-44f4-9d2e-470d3cba8011 admin admin] PATH_INFO: `/v3/roles` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:10:27.515977 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73cd3c41-a529-44f4-9d2e-470d3cba8011 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:10:27.518363 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73cd3c41-a529-44f4-9d2e-470d3cba8011 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:10:27.529068 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1329/2657] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Aug 30 02:10:27 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 02:10:27.533403 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7b2467e-781b-488f-96c8-6a1eea746171 admin admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:10:27.543189 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7b2467e-781b-488f-96c8-6a1eea746171 admin admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:10:27.547707 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57cbbaf4-8893-4a95-a190-d1d8adc6993c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:10:27.548526 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57cbbaf4-8893-4a95-a190-d1d8adc6993c admin admin] REQUEST_METHOD: `PUT` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:10:27.548738 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57cbbaf4-8893-4a95-a190-d1d8adc6993c admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:10:27.549009 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57cbbaf4-8893-4a95-a190-d1d8adc6993c admin admin] PATH_INFO: `/v3/projects/d085cdf435a445db82277b9465f1f5d5/users/549251884d8a4e5797cf9ac1af44dc57/roles/1fb81afcb9f245f3b989add94598a063` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:10:27.563184 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57cbbaf4-8893-4a95-a190-d1d8adc6993c admin admin] RBAC: Authorizing `identity:create_grant(project_id=d085cdf435a445db82277b9465f1f5d5, user_id=549251884d8a4e5797cf9ac1af44dc57, role_id=1fb81afcb9f245f3b989add94598a063)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:10:27.566354 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57cbbaf4-8893-4a95-a190-d1d8adc6993c admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:10:27.629539 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1329/2658] 10.222.0.27 () {60 vars in 1483 bytes} [Tue Aug 30 02:10:27 2022] PUT /identity/v3/projects/d085cdf435a445db82277b9465f1f5d5/users/549251884d8a4e5797cf9ac1af44dc57/roles/1fb81afcb9f245f3b989add94598a063 => generated 0 bytes in 97 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 02:10:27.641291 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-809738c4-fc2b-4fb1-9e36-ac856ab9743e None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:10:27.642279 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-809738c4-fc2b-4fb1-9e36-ac856ab9743e None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:10:27.642826 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-809738c4-fc2b-4fb1-9e36-ac856ab9743e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:10:27.670270 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-809738c4-fc2b-4fb1-9e36-ac856ab9743e None None] MFA Rules not processed for user `549251884d8a4e5797cf9ac1af44dc57`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:10:27.671479 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-809738c4-fc2b-4fb1-9e36-ac856ab9743e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:10:27.734312 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1330/2659] 10.222.0.27 () {58 vars in 958 bytes} [Tue Aug 30 02:10:27 2022] POST /identity/v3/auth/tokens => generated 3258 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 02:10:27.739449 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57cbbaf4-8893-4a95-a190-d1d8adc6993c admin admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:10:27.800142 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57cbbaf4-8893-4a95-a190-d1d8adc6993c admin admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:10:27.804692 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd5cf528-5547-45c3-8a2c-af29929ac014 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:10:27.805283 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd5cf528-5547-45c3-8a2c-af29929ac014 admin admin] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:10:27.805594 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd5cf528-5547-45c3-8a2c-af29929ac014 admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:10:27.805861 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd5cf528-5547-45c3-8a2c-af29929ac014 admin admin] PATH_INFO: `/v3/projects` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:10:27.806488 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd5cf528-5547-45c3-8a2c-af29929ac014 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:10:27.808347 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd5cf528-5547-45c3-8a2c-af29929ac014 admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:10:27.888689 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1330/2660] 10.222.0.27 () {60 vars in 1153 bytes} [Tue Aug 30 02:10:27 2022] POST /identity/v3/projects => generated 381 bytes in 150 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 02:10:27.892479 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-809738c4-fc2b-4fb1-9e36-ac856ab9743e None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:10:27.903075 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-809738c4-fc2b-4fb1-9e36-ac856ab9743e None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:10:27.947420 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:10:27.960029 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0cd85df-48d3-4fdb-aa77-c6729444ee99 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:10:27.960738 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0cd85df-48d3-4fdb-aa77-c6729444ee99 admin admin] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:10:27.961015 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0cd85df-48d3-4fdb-aa77-c6729444ee99 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:10:27.961282 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0cd85df-48d3-4fdb-aa77-c6729444ee99 admin admin] PATH_INFO: `/v3/users` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:10:27.961958 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0cd85df-48d3-4fdb-aa77-c6729444ee99 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:10:27.963827 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0cd85df-48d3-4fdb-aa77-c6729444ee99 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:10:28.057947 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1331/2661] 10.222.0.27 () {60 vars in 1144 bytes} [Tue Aug 30 02:10:27 2022] POST /identity/v3/users => generated 297 bytes in 166 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 30 02:10:28.062012 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd5cf528-5547-45c3-8a2c-af29929ac014 admin admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:10:28.073003 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd5cf528-5547-45c3-8a2c-af29929ac014 admin admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:10:28.077716 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d7b3eeb-71d2-4bb3-a2c0-d81e67a6c433 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:10:28.078344 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d7b3eeb-71d2-4bb3-a2c0-d81e67a6c433 admin admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:10:28.078626 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d7b3eeb-71d2-4bb3-a2c0-d81e67a6c433 admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:10:28.082450 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d7b3eeb-71d2-4bb3-a2c0-d81e67a6c433 admin admin] PATH_INFO: `/v3/roles` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:10:28.082450 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d7b3eeb-71d2-4bb3-a2c0-d81e67a6c433 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:10:28.082450 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d7b3eeb-71d2-4bb3-a2c0-d81e67a6c433 admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:10:28.092083 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1331/2662] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Aug 30 02:10:28 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 02:10:28.096861 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0cd85df-48d3-4fdb-aa77-c6729444ee99 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:10:28.107139 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0cd85df-48d3-4fdb-aa77-c6729444ee99 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:10:28.112340 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b60214a-d911-4585-9ebd-289924426247 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:10:28.112986 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b60214a-d911-4585-9ebd-289924426247 admin admin] REQUEST_METHOD: `PUT` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:10:28.113273 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b60214a-d911-4585-9ebd-289924426247 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:10:28.113579 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b60214a-d911-4585-9ebd-289924426247 admin admin] PATH_INFO: `/v3/projects/2a792650e35d4389bf474d9417bd6b61/users/533ad3c71868453587ed5c401dae01f0/roles/ead5fac86a0747ea8823a82efcea8ed7` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:10:28.128322 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b60214a-d911-4585-9ebd-289924426247 admin admin] RBAC: Authorizing `identity:create_grant(project_id=2a792650e35d4389bf474d9417bd6b61, user_id=533ad3c71868453587ed5c401dae01f0, role_id=ead5fac86a0747ea8823a82efcea8ed7)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:10:28.131155 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b60214a-d911-4585-9ebd-289924426247 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:10:28.204297 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1332/2663] 10.222.0.27 () {60 vars in 1483 bytes} [Tue Aug 30 02:10:28 2022] PUT /identity/v3/projects/2a792650e35d4389bf474d9417bd6b61/users/533ad3c71868453587ed5c401dae01f0/roles/ead5fac86a0747ea8823a82efcea8ed7 => generated 0 bytes in 108 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 02:10:28.208314 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d7b3eeb-71d2-4bb3-a2c0-d81e67a6c433 admin admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:10:28.218204 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d7b3eeb-71d2-4bb3-a2c0-d81e67a6c433 admin admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:10:28.275439 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa6d46cc-9d58-4443-96e0-fafac1e548e3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:10:28.276042 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa6d46cc-9d58-4443-96e0-fafac1e548e3 admin admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:10:28.276324 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa6d46cc-9d58-4443-96e0-fafac1e548e3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:10:28.276596 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa6d46cc-9d58-4443-96e0-fafac1e548e3 admin admin] PATH_INFO: `/v3/roles` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:10:28.277228 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa6d46cc-9d58-4443-96e0-fafac1e548e3 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:10:28.278847 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa6d46cc-9d58-4443-96e0-fafac1e548e3 admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:10:28.289241 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1332/2664] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Aug 30 02:10:28 2022] GET /identity/v3/roles => generated 1420 bytes in 81 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 02:10:28.293353 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b60214a-d911-4585-9ebd-289924426247 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:10:28.304127 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b60214a-d911-4585-9ebd-289924426247 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:10:28.308722 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea410a48-ebf9-40fe-9a47-e6126778b033 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:10:28.309356 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea410a48-ebf9-40fe-9a47-e6126778b033 admin admin] REQUEST_METHOD: `PUT` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:10:28.309668 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea410a48-ebf9-40fe-9a47-e6126778b033 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:10:28.309943 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea410a48-ebf9-40fe-9a47-e6126778b033 admin admin] PATH_INFO: `/v3/projects/2a792650e35d4389bf474d9417bd6b61/users/533ad3c71868453587ed5c401dae01f0/roles/ead5fac86a0747ea8823a82efcea8ed7` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:10:28.311834 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea410a48-ebf9-40fe-9a47-e6126778b033 admin admin] RBAC: Authorizing `identity:create_grant(project_id=2a792650e35d4389bf474d9417bd6b61, user_id=533ad3c71868453587ed5c401dae01f0, role_id=ead5fac86a0747ea8823a82efcea8ed7)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:10:28.314925 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea410a48-ebf9-40fe-9a47-e6126778b033 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:10:28.324958 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1333/2665] 10.222.0.27 () {60 vars in 1483 bytes} [Tue Aug 30 02:10:28 2022] PUT /identity/v3/projects/2a792650e35d4389bf474d9417bd6b61/users/533ad3c71868453587ed5c401dae01f0/roles/ead5fac86a0747ea8823a82efcea8ed7 => generated 0 bytes in 32 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 02:10:28.335231 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14e5d8f3-4c10-46d5-9431-9d3490e54d15 None None] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:10:28.335518 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14e5d8f3-4c10-46d5-9431-9d3490e54d15 None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:10:28.335788 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14e5d8f3-4c10-46d5-9431-9d3490e54d15 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:10:28.364322 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.auth.core [None req-14e5d8f3-4c10-46d5-9431-9d3490e54d15 None None] MFA Rules not processed for user `533ad3c71868453587ed5c401dae01f0`. Rule list: `[]` (Enabled: `True`). {{(pid=81578) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:10:28.365544 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-14e5d8f3-4c10-46d5-9431-9d3490e54d15 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:10:28.398269 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:10:28.432204 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1333/2666] 10.222.0.27 () {58 vars in 958 bytes} [Tue Aug 30 02:10:28 2022] POST /identity/v3/auth/tokens => generated 3325 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 02:10:28.443266 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea410a48-ebf9-40fe-9a47-e6126778b033 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:10:28.444959 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-ea410a48-ebf9-40fe-9a47-e6126778b033 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:10:28.506394 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea410a48-ebf9-40fe-9a47-e6126778b033 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:10:28.510054 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca1ae7c7-efd1-40e8-94c6-9f0757ce794c service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ab936b3767a8417aa80c8c8f1b84b667', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4562f3951fbe465280073e60a11e44e2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:10:28.510659 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca1ae7c7-efd1-40e8-94c6-9f0757ce794c service cinder] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:10:28.510938 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca1ae7c7-efd1-40e8-94c6-9f0757ce794c service cinder] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:10:28.511206 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca1ae7c7-efd1-40e8-94c6-9f0757ce794c service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:10:28.520465 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca1ae7c7-efd1-40e8-94c6-9f0757ce794c service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:10:28.521937 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca1ae7c7-efd1-40e8-94c6-9f0757ce794c service cinder] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:10:28.531150 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1334/2667] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Aug 30 02:10:28 2022] GET /identity/v3/auth/tokens => generated 3325 bytes in 88 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 02:10:28.636283 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14e5d8f3-4c10-46d5-9431-9d3490e54d15 None None] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:10:28.647041 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14e5d8f3-4c10-46d5-9431-9d3490e54d15 None None] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:10:28.650617 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a670a403-84ce-4fa2-a116-a3ee2425d136 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ab936b3767a8417aa80c8c8f1b84b667', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4562f3951fbe465280073e60a11e44e2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:10:28.651210 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a670a403-84ce-4fa2-a116-a3ee2425d136 service cinder] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:10:28.651491 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a670a403-84ce-4fa2-a116-a3ee2425d136 service cinder] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:10:28.651761 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a670a403-84ce-4fa2-a116-a3ee2425d136 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:10:28.709445 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a670a403-84ce-4fa2-a116-a3ee2425d136 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:10:28.710903 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a670a403-84ce-4fa2-a116-a3ee2425d136 service cinder] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:10:28.720172 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1334/2668] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Aug 30 02:10:28 2022] GET /identity/v3/auth/tokens => generated 3258 bytes in 84 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 02:10:29.610948 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c71bc8b-0fe6-4c15-97f3-0be8e42dc298 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:10:29.612039 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c71bc8b-0fe6-4c15-97f3-0be8e42dc298 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:10:29.612159 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c71bc8b-0fe6-4c15-97f3-0be8e42dc298 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:10:29.631301 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-2c71bc8b-0fe6-4c15-97f3-0be8e42dc298 None None] MFA Rules not processed for user `121c374d49a64335ab74a431cfbc1320`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:10:29.632479 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-2c71bc8b-0fe6-4c15-97f3-0be8e42dc298 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:10:29.686887 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1335/2669] 10.222.0.27 () {58 vars in 1009 bytes} [Tue Aug 30 02:10:29 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 77 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 02:10:29.708802 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a670a403-84ce-4fa2-a116-a3ee2425d136 service cinder] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:10:29.710265 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-a670a403-84ce-4fa2-a116-a3ee2425d136 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:10:29.780576 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a670a403-84ce-4fa2-a116-a3ee2425d136 service cinder] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:10:29.786040 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb77f44c-b32a-4635-ab0b-321e4da07f0f service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b4f0008fde424f22bc1f7307cf422774', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4562f3951fbe465280073e60a11e44e2', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:10:29.786644 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb77f44c-b32a-4635-ab0b-321e4da07f0f service swift] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:10:29.786920 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb77f44c-b32a-4635-ab0b-321e4da07f0f service swift] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:10:29.787189 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb77f44c-b32a-4635-ab0b-321e4da07f0f service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:10:29.794994 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb77f44c-b32a-4635-ab0b-321e4da07f0f service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:10:29.796536 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb77f44c-b32a-4635-ab0b-321e4da07f0f service swift] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:10:29.805282 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1335/2670] 10.222.0.27 () {60 vars in 1384 bytes} [Tue Aug 30 02:10:29 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 97 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Aug 30 02:10:31.868166 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c71bc8b-0fe6-4c15-97f3-0be8e42dc298 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:10:31.928940 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:10:31.953674 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c71bc8b-0fe6-4c15-97f3-0be8e42dc298 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:10:31.959522 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e95cd981-ea7c-49c5-b394-28c6325a89f4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:10:31.961683 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e95cd981-ea7c-49c5-b394-28c6325a89f4 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:10:31.961683 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e95cd981-ea7c-49c5-b394-28c6325a89f4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:10:31.961683 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e95cd981-ea7c-49c5-b394-28c6325a89f4 admin admin] PATH_INFO: `/v3/users/549251884d8a4e5797cf9ac1af44dc57` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:10:31.963050 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e95cd981-ea7c-49c5-b394-28c6325a89f4 admin admin] RBAC: Authorizing `identity:delete_user(user_id=549251884d8a4e5797cf9ac1af44dc57)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:10:31.965855 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e95cd981-ea7c-49c5-b394-28c6325a89f4 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:10:32.186412 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-e95cd981-ea7c-49c5-b394-28c6325a89f4 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '549251884d8a4e5797cf9ac1af44dc57'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:10:32.187117 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-e95cd981-ea7c-49c5-b394-28c6325a89f4 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '549251884d8a4e5797cf9ac1af44dc57'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:10:32.187736 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-e95cd981-ea7c-49c5-b394-28c6325a89f4 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '549251884d8a4e5797cf9ac1af44dc57'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:10:32.202190 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-e95cd981-ea7c-49c5-b394-28c6325a89f4 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '549251884d8a4e5797cf9ac1af44dc57'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:10:32.210421 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-e95cd981-ea7c-49c5-b394-28c6325a89f4 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '549251884d8a4e5797cf9ac1af44dc57'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:10:32.279381 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1336/2671] 10.222.0.27 () {58 vars in 1224 bytes} [Tue Aug 30 02:10:31 2022] DELETE /identity/v3/users/549251884d8a4e5797cf9ac1af44dc57 => generated 0 bytes in 411 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 02:10:32.284976 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d15144b-9c82-4cbf-9b5e-d74ff5499df6 None None] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:10:32.289879 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d15144b-9c82-4cbf-9b5e-d74ff5499df6 None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:10:32.289879 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d15144b-9c82-4cbf-9b5e-d74ff5499df6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:10:32.306959 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.auth.core [None req-0d15144b-9c82-4cbf-9b5e-d74ff5499df6 None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81578) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:10:32.308311 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-0d15144b-9c82-4cbf-9b5e-d74ff5499df6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:10:32.377594 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:10:32.384088 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1336/2672] 10.222.0.27 () {58 vars in 958 bytes} [Tue Aug 30 02:10:32 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 02:10:32.395053 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e95cd981-ea7c-49c5-b394-28c6325a89f4 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:10:32.396522 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-e95cd981-ea7c-49c5-b394-28c6325a89f4 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:10:32.470172 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e95cd981-ea7c-49c5-b394-28c6325a89f4 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:10:32.474810 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e473085c-6a31-4710-b950-f82cd15e9be2 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f3ea1580c56f4814a92bbb2acf542905', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4562f3951fbe465280073e60a11e44e2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:10:32.475465 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e473085c-6a31-4710-b950-f82cd15e9be2 service neutron] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:10:32.475780 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e473085c-6a31-4710-b950-f82cd15e9be2 service neutron] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:10:32.476052 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e473085c-6a31-4710-b950-f82cd15e9be2 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:10:32.487837 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e473085c-6a31-4710-b950-f82cd15e9be2 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:10:32.489280 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e473085c-6a31-4710-b950-f82cd15e9be2 service neutron] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:10:32.500248 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1337/2673] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Aug 30 02:10:32 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 106 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 02:10:33.043969 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d15144b-9c82-4cbf-9b5e-d74ff5499df6 None None] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:10:33.045221 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-0d15144b-9c82-4cbf-9b5e-d74ff5499df6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:10:33.076581 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d15144b-9c82-4cbf-9b5e-d74ff5499df6 None None] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:10:33.089794 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b8dfb12-5479-4b2b-aaf9-7ba775cf087e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:10:33.090593 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b8dfb12-5479-4b2b-aaf9-7ba775cf087e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:10:33.091024 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b8dfb12-5479-4b2b-aaf9-7ba775cf087e admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:10:33.091362 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b8dfb12-5479-4b2b-aaf9-7ba775cf087e admin admin] PATH_INFO: `/v3/projects/d085cdf435a445db82277b9465f1f5d5` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:10:33.092751 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b8dfb12-5479-4b2b-aaf9-7ba775cf087e admin admin] RBAC: Authorizing `identity:delete_project(project_id=d085cdf435a445db82277b9465f1f5d5)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:10:33.094888 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b8dfb12-5479-4b2b-aaf9-7ba775cf087e admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:10:33.234858 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-5b8dfb12-5479-4b2b-aaf9-7ba775cf087e admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'd085cdf435a445db82277b9465f1f5d5', 'request_id': 'req-5b8dfb12-5479-4b2b-aaf9-7ba775cf087e'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:10:33.243157 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-5b8dfb12-5479-4b2b-aaf9-7ba775cf087e admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'd085cdf435a445db82277b9465f1f5d5', 'request_id': 'req-5b8dfb12-5479-4b2b-aaf9-7ba775cf087e'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:10:33.254596 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-5b8dfb12-5479-4b2b-aaf9-7ba775cf087e admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'd085cdf435a445db82277b9465f1f5d5', 'request_id': 'req-5b8dfb12-5479-4b2b-aaf9-7ba775cf087e'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:10:33.347800 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-5b8dfb12-5479-4b2b-aaf9-7ba775cf087e admin admin] The token cache is being invalidate because project d085cdf435a445db82277b9465f1f5d5 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81578) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 30 02:10:33.348507 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-5b8dfb12-5479-4b2b-aaf9-7ba775cf087e admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:10:33.349150 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-5b8dfb12-5479-4b2b-aaf9-7ba775cf087e admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:10:33.352022 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1337/2674] 10.222.0.27 () {58 vars in 1233 bytes} [Tue Aug 30 02:10:33 2022] DELETE /identity/v3/projects/d085cdf435a445db82277b9465f1f5d5 => generated 0 bytes in 309 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 02:10:33.357970 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e473085c-6a31-4710-b950-f82cd15e9be2 service neutron] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:10:33.372701 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e473085c-6a31-4710-b950-f82cd15e9be2 service neutron] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:10:33.440840 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c350b53f-eb10-4aab-a82a-1eb2834c8aaa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:10:33.441487 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c350b53f-eb10-4aab-a82a-1eb2834c8aaa admin admin] REQUEST_METHOD: `DELETE` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:10:33.441808 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c350b53f-eb10-4aab-a82a-1eb2834c8aaa admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:10:33.442078 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c350b53f-eb10-4aab-a82a-1eb2834c8aaa admin admin] PATH_INFO: `/v3/users/533ad3c71868453587ed5c401dae01f0` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:10:33.443234 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c350b53f-eb10-4aab-a82a-1eb2834c8aaa admin admin] RBAC: Authorizing `identity:delete_user(user_id=533ad3c71868453587ed5c401dae01f0)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:10:33.445156 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c350b53f-eb10-4aab-a82a-1eb2834c8aaa admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:10:33.459210 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:10:33.708840 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-c350b53f-eb10-4aab-a82a-1eb2834c8aaa admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '533ad3c71868453587ed5c401dae01f0'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:10:33.709571 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-c350b53f-eb10-4aab-a82a-1eb2834c8aaa admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '533ad3c71868453587ed5c401dae01f0'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:10:33.710185 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-c350b53f-eb10-4aab-a82a-1eb2834c8aaa admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '533ad3c71868453587ed5c401dae01f0'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:10:33.730436 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-c350b53f-eb10-4aab-a82a-1eb2834c8aaa admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '533ad3c71868453587ed5c401dae01f0'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:10:33.738049 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-c350b53f-eb10-4aab-a82a-1eb2834c8aaa admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '533ad3c71868453587ed5c401dae01f0'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:10:33.808321 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1338/2675] 10.222.0.27 () {58 vars in 1224 bytes} [Tue Aug 30 02:10:33 2022] DELETE /identity/v3/users/533ad3c71868453587ed5c401dae01f0 => generated 0 bytes in 451 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 02:10:34.313320 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b8dfb12-5479-4b2b-aaf9-7ba775cf087e admin admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:10:34.316280 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-5b8dfb12-5479-4b2b-aaf9-7ba775cf087e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:10:34.331001 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b8dfb12-5479-4b2b-aaf9-7ba775cf087e admin admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:10:34.423323 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a59fdb54-f0a5-4ee1-81b4-efa66660e82f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:10:34.424124 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a59fdb54-f0a5-4ee1-81b4-efa66660e82f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:10:34.424538 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a59fdb54-f0a5-4ee1-81b4-efa66660e82f admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:10:34.424874 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a59fdb54-f0a5-4ee1-81b4-efa66660e82f admin admin] PATH_INFO: `/v3/projects/2a792650e35d4389bf474d9417bd6b61` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:10:34.426109 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a59fdb54-f0a5-4ee1-81b4-efa66660e82f admin admin] RBAC: Authorizing `identity:delete_project(project_id=2a792650e35d4389bf474d9417bd6b61)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:10:34.428467 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a59fdb54-f0a5-4ee1-81b4-efa66660e82f admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:10:34.535594 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-a59fdb54-f0a5-4ee1-81b4-efa66660e82f admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '2a792650e35d4389bf474d9417bd6b61', 'request_id': 'req-a59fdb54-f0a5-4ee1-81b4-efa66660e82f'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:10:34.541309 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-a59fdb54-f0a5-4ee1-81b4-efa66660e82f admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '2a792650e35d4389bf474d9417bd6b61', 'request_id': 'req-a59fdb54-f0a5-4ee1-81b4-efa66660e82f'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:10:34.549190 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-a59fdb54-f0a5-4ee1-81b4-efa66660e82f admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '2a792650e35d4389bf474d9417bd6b61', 'request_id': 'req-a59fdb54-f0a5-4ee1-81b4-efa66660e82f'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:10:34.630307 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-a59fdb54-f0a5-4ee1-81b4-efa66660e82f admin admin] The token cache is being invalidate because project 2a792650e35d4389bf474d9417bd6b61 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81578) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 30 02:10:34.630307 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-a59fdb54-f0a5-4ee1-81b4-efa66660e82f admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:10:34.630936 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-a59fdb54-f0a5-4ee1-81b4-efa66660e82f admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:10:34.632492 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1338/2676] 10.222.0.27 () {58 vars in 1233 bytes} [Tue Aug 30 02:10:34 2022] DELETE /identity/v3/projects/2a792650e35d4389bf474d9417bd6b61 => generated 0 bytes in 320 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 30 02:10:34.637572 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d2eda0c-8034-48dd-ac62-de6d3741bf39 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:10:34.638257 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d2eda0c-8034-48dd-ac62-de6d3741bf39 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:10:34.638257 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d2eda0c-8034-48dd-ac62-de6d3741bf39 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:10:34.660297 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-8d2eda0c-8034-48dd-ac62-de6d3741bf39 None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:10:34.663341 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-8d2eda0c-8034-48dd-ac62-de6d3741bf39 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:10:34.745031 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1339/2677] 10.222.0.27 () {58 vars in 958 bytes} [Tue Aug 30 02:10:34 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 02:10:34.754782 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abb48ddb-f298-437b-9c91-100ec6624171 None None] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:10:34.755086 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abb48ddb-f298-437b-9c91-100ec6624171 None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:10:34.755337 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abb48ddb-f298-437b-9c91-100ec6624171 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:10:34.776758 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.auth.core [None req-abb48ddb-f298-437b-9c91-100ec6624171 None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81578) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:10:34.778847 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-abb48ddb-f298-437b-9c91-100ec6624171 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:10:34.789611 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1339/2678] 10.222.0.27 () {58 vars in 958 bytes} [Tue Aug 30 02:10:34 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 36 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 30 02:10:34.793953 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d2eda0c-8034-48dd-ac62-de6d3741bf39 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:10:34.808870 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d2eda0c-8034-48dd-ac62-de6d3741bf39 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:10:34.816393 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-764bf080-5b70-49f1-b5ab-66a5433ce5f0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:10:34.817582 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-764bf080-5b70-49f1-b5ab-66a5433ce5f0 admin admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:10:34.817987 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-764bf080-5b70-49f1-b5ab-66a5433ce5f0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:10:34.818357 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-764bf080-5b70-49f1-b5ab-66a5433ce5f0 admin admin] PATH_INFO: `/v3/domains` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:10:34.819250 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-764bf080-5b70-49f1-b5ab-66a5433ce5f0 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 02:10:34.819688 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-764bf080-5b70-49f1-b5ab-66a5433ce5f0 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:10:34.821745 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-764bf080-5b70-49f1-b5ab-66a5433ce5f0 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:10:34.846876 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abb48ddb-f298-437b-9c91-100ec6624171 None None] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:10:34.847354 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1340/2679] 10.222.0.27 () {58 vars in 1153 bytes} [Tue Aug 30 02:10:34 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 53 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 02:10:34.857495 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abb48ddb-f298-437b-9c91-100ec6624171 None None] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:10:34.865151 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92a37198-e2a8-4c26-a7b9-da3a3743ede1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:10:34.865814 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92a37198-e2a8-4c26-a7b9-da3a3743ede1 admin admin] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:10:34.866094 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92a37198-e2a8-4c26-a7b9-da3a3743ede1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:10:34.866415 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92a37198-e2a8-4c26-a7b9-da3a3743ede1 admin admin] PATH_INFO: `/v3/projects` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:10:34.867141 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92a37198-e2a8-4c26-a7b9-da3a3743ede1 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:10:34.869140 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92a37198-e2a8-4c26-a7b9-da3a3743ede1 admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:10:34.874374 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=84,keystone:DELETE=14,keystone:INSERT=2 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:10:34.974215 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-764bf080-5b70-49f1-b5ab-66a5433ce5f0 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:10:34.982521 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1340/2680] 10.222.0.27 () {60 vars in 1153 bytes} [Tue Aug 30 02:10:34 2022] POST /identity/v3/projects => generated 381 bytes in 134 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 02:10:34.991430 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-764bf080-5b70-49f1-b5ab-66a5433ce5f0 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:10:35.077156 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a245f5c0-c3ca-4ce7-bb3e-5dfe412b3693 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:10:35.078161 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a245f5c0-c3ca-4ce7-bb3e-5dfe412b3693 admin admin] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:10:35.078532 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a245f5c0-c3ca-4ce7-bb3e-5dfe412b3693 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:10:35.078944 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a245f5c0-c3ca-4ce7-bb3e-5dfe412b3693 admin admin] PATH_INFO: `/v3/users` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:10:35.080628 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a245f5c0-c3ca-4ce7-bb3e-5dfe412b3693 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:10:35.082879 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a245f5c0-c3ca-4ce7-bb3e-5dfe412b3693 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:10:35.190079 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1341/2681] 10.222.0.27 () {60 vars in 1144 bytes} [Tue Aug 30 02:10:34 2022] POST /identity/v3/users => generated 291 bytes in 216 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 02:10:35.194390 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92a37198-e2a8-4c26-a7b9-da3a3743ede1 admin admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:10:35.208875 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92a37198-e2a8-4c26-a7b9-da3a3743ede1 admin admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:10:35.213801 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d757b68f-0bdb-452d-8328-df4cd18f989e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:10:35.214329 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d757b68f-0bdb-452d-8328-df4cd18f989e admin admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:10:35.214609 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d757b68f-0bdb-452d-8328-df4cd18f989e admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:10:35.214880 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d757b68f-0bdb-452d-8328-df4cd18f989e admin admin] PATH_INFO: `/v3/roles` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:10:35.215655 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d757b68f-0bdb-452d-8328-df4cd18f989e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:10:35.217258 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d757b68f-0bdb-452d-8328-df4cd18f989e admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:10:35.229493 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1341/2682] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Aug 30 02:10:35 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 30 02:10:35.234171 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a245f5c0-c3ca-4ce7-bb3e-5dfe412b3693 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:10:35.242962 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:DELETE=10,keystone:SELECT=86,keystone:INSERT=4 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:10:35.247678 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a245f5c0-c3ca-4ce7-bb3e-5dfe412b3693 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:10:35.253132 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0baca9e1-034f-4ada-beb3-b6f3ba906bf2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:10:35.255581 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0baca9e1-034f-4ada-beb3-b6f3ba906bf2 admin admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:10:35.255581 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0baca9e1-034f-4ada-beb3-b6f3ba906bf2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:10:35.255581 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0baca9e1-034f-4ada-beb3-b6f3ba906bf2 admin admin] PATH_INFO: `/v3/roles` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:10:35.255581 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0baca9e1-034f-4ada-beb3-b6f3ba906bf2 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:10:35.256843 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0baca9e1-034f-4ada-beb3-b6f3ba906bf2 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:10:35.272273 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1342/2683] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Aug 30 02:10:35 2022] GET /identity/v3/roles => generated 1420 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 02:10:35.277586 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d757b68f-0bdb-452d-8328-df4cd18f989e admin admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:10:35.289585 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d757b68f-0bdb-452d-8328-df4cd18f989e admin admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:10:35.295676 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-629a869f-58e5-4be2-b468-eaacfc150112 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:10:35.296581 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-629a869f-58e5-4be2-b468-eaacfc150112 admin admin] REQUEST_METHOD: `PUT` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:10:35.296860 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-629a869f-58e5-4be2-b468-eaacfc150112 admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:10:35.297138 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-629a869f-58e5-4be2-b468-eaacfc150112 admin admin] PATH_INFO: `/v3/projects/7efd3cfdc43c4c9aad8ca2e72c723069/users/d597e2969de44ab382d467e1998c4c5d/roles/1fb81afcb9f245f3b989add94598a063` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:10:35.315064 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-629a869f-58e5-4be2-b468-eaacfc150112 admin admin] RBAC: Authorizing `identity:create_grant(project_id=7efd3cfdc43c4c9aad8ca2e72c723069, user_id=d597e2969de44ab382d467e1998c4c5d, role_id=1fb81afcb9f245f3b989add94598a063)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:10:35.317646 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-629a869f-58e5-4be2-b468-eaacfc150112 admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:10:35.390731 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1342/2684] 10.222.0.27 () {60 vars in 1483 bytes} [Tue Aug 30 02:10:35 2022] PUT /identity/v3/projects/7efd3cfdc43c4c9aad8ca2e72c723069/users/d597e2969de44ab382d467e1998c4c5d/roles/1fb81afcb9f245f3b989add94598a063 => generated 0 bytes in 114 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 30 02:10:35.403277 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8099af20-de03-4448-a393-19b2e0a0eacc None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:10:35.403851 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8099af20-de03-4448-a393-19b2e0a0eacc None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:10:35.404137 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8099af20-de03-4448-a393-19b2e0a0eacc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:10:35.438601 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-8099af20-de03-4448-a393-19b2e0a0eacc None None] MFA Rules not processed for user `d597e2969de44ab382d467e1998c4c5d`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:10:35.440593 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-8099af20-de03-4448-a393-19b2e0a0eacc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:10:35.522542 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1343/2685] 10.222.0.27 () {58 vars in 958 bytes} [Tue Aug 30 02:10:35 2022] POST /identity/v3/auth/tokens => generated 3258 bytes in 120 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 02:10:35.527266 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-629a869f-58e5-4be2-b468-eaacfc150112 admin admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:10:35.600561 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-629a869f-58e5-4be2-b468-eaacfc150112 admin admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:10:35.605591 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9da37bf-bcf0-4952-9a12-ee61e04d7525 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:10:35.606207 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9da37bf-bcf0-4952-9a12-ee61e04d7525 admin admin] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:10:35.606481 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9da37bf-bcf0-4952-9a12-ee61e04d7525 admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:10:35.606791 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9da37bf-bcf0-4952-9a12-ee61e04d7525 admin admin] PATH_INFO: `/v3/projects` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:10:35.607497 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9da37bf-bcf0-4952-9a12-ee61e04d7525 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:10:35.609601 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9da37bf-bcf0-4952-9a12-ee61e04d7525 admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:10:35.894578 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1343/2686] 10.222.0.27 () {60 vars in 1153 bytes} [Tue Aug 30 02:10:35 2022] POST /identity/v3/projects => generated 381 bytes in 368 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 02:10:35.898790 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8099af20-de03-4448-a393-19b2e0a0eacc None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:10:35.909360 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8099af20-de03-4448-a393-19b2e0a0eacc None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:10:35.971555 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82412da7-882d-4682-bfd6-fdcc48e51a86 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:10:35.972316 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82412da7-882d-4682-bfd6-fdcc48e51a86 admin admin] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:10:35.972529 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82412da7-882d-4682-bfd6-fdcc48e51a86 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:10:35.972798 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82412da7-882d-4682-bfd6-fdcc48e51a86 admin admin] PATH_INFO: `/v3/users` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:10:35.973698 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82412da7-882d-4682-bfd6-fdcc48e51a86 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:10:35.975574 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82412da7-882d-4682-bfd6-fdcc48e51a86 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:10:36.087633 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1344/2687] 10.222.0.27 () {60 vars in 1144 bytes} [Tue Aug 30 02:10:35 2022] POST /identity/v3/users => generated 297 bytes in 189 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 02:10:36.091836 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9da37bf-bcf0-4952-9a12-ee61e04d7525 admin admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:10:36.105046 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9da37bf-bcf0-4952-9a12-ee61e04d7525 admin admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:10:36.110358 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a458521b-710c-4f32-814e-db7c2cc8f95b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:10:36.111608 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a458521b-710c-4f32-814e-db7c2cc8f95b admin admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:10:36.112012 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a458521b-710c-4f32-814e-db7c2cc8f95b admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:10:36.112293 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a458521b-710c-4f32-814e-db7c2cc8f95b admin admin] PATH_INFO: `/v3/roles` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:10:36.113138 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a458521b-710c-4f32-814e-db7c2cc8f95b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:10:36.114870 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a458521b-710c-4f32-814e-db7c2cc8f95b admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:10:36.134939 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1344/2688] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Aug 30 02:10:36 2022] GET /identity/v3/roles => generated 1420 bytes in 43 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 02:10:36.138893 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82412da7-882d-4682-bfd6-fdcc48e51a86 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:10:36.150196 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82412da7-882d-4682-bfd6-fdcc48e51a86 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:10:36.155791 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac68ea7b-0b2f-4bf2-9354-6fa2cdacf8cb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:10:36.160150 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac68ea7b-0b2f-4bf2-9354-6fa2cdacf8cb admin admin] REQUEST_METHOD: `PUT` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:10:36.160150 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac68ea7b-0b2f-4bf2-9354-6fa2cdacf8cb admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:10:36.160150 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac68ea7b-0b2f-4bf2-9354-6fa2cdacf8cb admin admin] PATH_INFO: `/v3/projects/c2b165dfcd5b41818cce106bc8e0c886/users/bc0c1737e6b648428f06ebc8773f28ea/roles/ead5fac86a0747ea8823a82efcea8ed7` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:10:36.177342 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac68ea7b-0b2f-4bf2-9354-6fa2cdacf8cb admin admin] RBAC: Authorizing `identity:create_grant(project_id=c2b165dfcd5b41818cce106bc8e0c886, user_id=bc0c1737e6b648428f06ebc8773f28ea, role_id=ead5fac86a0747ea8823a82efcea8ed7)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:10:36.180738 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac68ea7b-0b2f-4bf2-9354-6fa2cdacf8cb admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:10:36.271980 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1345/2689] 10.222.0.27 () {60 vars in 1483 bytes} [Tue Aug 30 02:10:36 2022] PUT /identity/v3/projects/c2b165dfcd5b41818cce106bc8e0c886/users/bc0c1737e6b648428f06ebc8773f28ea/roles/ead5fac86a0747ea8823a82efcea8ed7 => generated 0 bytes in 133 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 02:10:36.276307 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a458521b-710c-4f32-814e-db7c2cc8f95b admin admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:10:36.289297 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a458521b-710c-4f32-814e-db7c2cc8f95b admin admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:10:36.386983 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae90433d-f3ac-4856-a4c2-29090b87d4a6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:10:36.387788 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae90433d-f3ac-4856-a4c2-29090b87d4a6 admin admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:10:36.388157 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae90433d-f3ac-4856-a4c2-29090b87d4a6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:10:36.388560 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae90433d-f3ac-4856-a4c2-29090b87d4a6 admin admin] PATH_INFO: `/v3/roles` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:10:36.389436 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae90433d-f3ac-4856-a4c2-29090b87d4a6 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:10:36.391045 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae90433d-f3ac-4856-a4c2-29090b87d4a6 admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:10:36.405512 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1345/2690] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Aug 30 02:10:36 2022] GET /identity/v3/roles => generated 1420 bytes in 127 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 02:10:36.407114 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac68ea7b-0b2f-4bf2-9354-6fa2cdacf8cb admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:10:36.421356 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac68ea7b-0b2f-4bf2-9354-6fa2cdacf8cb admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:10:36.427684 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfcedbba-668a-4bb0-95d8-e8ab28739da5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:10:36.428405 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfcedbba-668a-4bb0-95d8-e8ab28739da5 admin admin] REQUEST_METHOD: `PUT` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:10:36.428781 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfcedbba-668a-4bb0-95d8-e8ab28739da5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:10:36.429197 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfcedbba-668a-4bb0-95d8-e8ab28739da5 admin admin] PATH_INFO: `/v3/projects/c2b165dfcd5b41818cce106bc8e0c886/users/bc0c1737e6b648428f06ebc8773f28ea/roles/ead5fac86a0747ea8823a82efcea8ed7` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:10:36.431279 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dfcedbba-668a-4bb0-95d8-e8ab28739da5 admin admin] RBAC: Authorizing `identity:create_grant(project_id=c2b165dfcd5b41818cce106bc8e0c886, user_id=bc0c1737e6b648428f06ebc8773f28ea, role_id=ead5fac86a0747ea8823a82efcea8ed7)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:10:36.433880 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dfcedbba-668a-4bb0-95d8-e8ab28739da5 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:10:36.446244 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1346/2691] 10.222.0.27 () {60 vars in 1483 bytes} [Tue Aug 30 02:10:36 2022] PUT /identity/v3/projects/c2b165dfcd5b41818cce106bc8e0c886/users/bc0c1737e6b648428f06ebc8773f28ea/roles/ead5fac86a0747ea8823a82efcea8ed7 => generated 0 bytes in 40 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 02:10:36.457066 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daa71cef-d4e6-43f3-baef-569fad4fb035 None None] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:10:36.457376 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daa71cef-d4e6-43f3-baef-569fad4fb035 None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:10:36.457904 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daa71cef-d4e6-43f3-baef-569fad4fb035 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:10:36.491309 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.auth.core [None req-daa71cef-d4e6-43f3-baef-569fad4fb035 None None] MFA Rules not processed for user `bc0c1737e6b648428f06ebc8773f28ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81578) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:10:36.492654 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-daa71cef-d4e6-43f3-baef-569fad4fb035 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:10:36.505113 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:10:36.579454 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1346/2692] 10.222.0.27 () {58 vars in 958 bytes} [Tue Aug 30 02:10:36 2022] POST /identity/v3/auth/tokens => generated 3325 bytes in 123 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 02:10:36.593384 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfcedbba-668a-4bb0-95d8-e8ab28739da5 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:10:36.595737 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-dfcedbba-668a-4bb0-95d8-e8ab28739da5 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:10:36.649396 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:10:36.673667 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfcedbba-668a-4bb0-95d8-e8ab28739da5 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:10:36.678338 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52859570-1eb4-48dd-9a22-04cf4a6d0dcc service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ab936b3767a8417aa80c8c8f1b84b667', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4562f3951fbe465280073e60a11e44e2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:10:36.679111 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52859570-1eb4-48dd-9a22-04cf4a6d0dcc service cinder] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:10:36.679513 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52859570-1eb4-48dd-9a22-04cf4a6d0dcc service cinder] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:10:36.680099 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52859570-1eb4-48dd-9a22-04cf4a6d0dcc service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:10:36.691455 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52859570-1eb4-48dd-9a22-04cf4a6d0dcc service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:10:36.693256 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52859570-1eb4-48dd-9a22-04cf4a6d0dcc service cinder] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:10:36.710405 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1347/2693] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Aug 30 02:10:36 2022] GET /identity/v3/auth/tokens => generated 3325 bytes in 117 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 30 02:10:37.297391 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-daa71cef-d4e6-43f3-baef-569fad4fb035 None None] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:10:37.375643 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-daa71cef-d4e6-43f3-baef-569fad4fb035 None None] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:10:37.380979 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35a20626-50ae-4403-b094-abde170693f7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:10:37.381652 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35a20626-50ae-4403-b094-abde170693f7 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:10:37.381933 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35a20626-50ae-4403-b094-abde170693f7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:10:37.382481 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35a20626-50ae-4403-b094-abde170693f7 admin admin] PATH_INFO: `/v3/users/d597e2969de44ab382d467e1998c4c5d` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:10:37.383645 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35a20626-50ae-4403-b094-abde170693f7 admin admin] RBAC: Authorizing `identity:delete_user(user_id=d597e2969de44ab382d467e1998c4c5d)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:10:37.385690 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35a20626-50ae-4403-b094-abde170693f7 admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:10:37.581434 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-35a20626-50ae-4403-b094-abde170693f7 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'd597e2969de44ab382d467e1998c4c5d'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:10:37.582134 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-35a20626-50ae-4403-b094-abde170693f7 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'd597e2969de44ab382d467e1998c4c5d'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:10:37.650370 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-35a20626-50ae-4403-b094-abde170693f7 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'd597e2969de44ab382d467e1998c4c5d'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:10:37.650995 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-35a20626-50ae-4403-b094-abde170693f7 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'd597e2969de44ab382d467e1998c4c5d'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:10:37.662918 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-35a20626-50ae-4403-b094-abde170693f7 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'd597e2969de44ab382d467e1998c4c5d'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:10:37.684270 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1347/2694] 10.222.0.27 () {58 vars in 1224 bytes} [Tue Aug 30 02:10:37 2022] DELETE /identity/v3/users/d597e2969de44ab382d467e1998c4c5d => generated 0 bytes in 389 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 02:10:37.688832 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9524708-36ac-4afb-92e0-112cde22172e None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:10:37.690022 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9524708-36ac-4afb-92e0-112cde22172e None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:10:37.690022 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9524708-36ac-4afb-92e0-112cde22172e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:10:37.712267 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-d9524708-36ac-4afb-92e0-112cde22172e None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:10:37.713704 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-d9524708-36ac-4afb-92e0-112cde22172e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:10:37.785639 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1348/2695] 10.222.0.27 () {58 vars in 958 bytes} [Tue Aug 30 02:10:37 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 30 02:10:37.795477 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35a20626-50ae-4403-b094-abde170693f7 admin admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:10:37.797313 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-35a20626-50ae-4403-b094-abde170693f7 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:10:37.829201 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:10:37.862000 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35a20626-50ae-4403-b094-abde170693f7 admin admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:10:37.867732 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c413c57-ac77-418b-bbe2-89c5d1b9f726 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f3ea1580c56f4814a92bbb2acf542905', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4562f3951fbe465280073e60a11e44e2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:10:37.867732 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c413c57-ac77-418b-bbe2-89c5d1b9f726 service neutron] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:10:37.867732 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c413c57-ac77-418b-bbe2-89c5d1b9f726 service neutron] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:10:37.867732 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c413c57-ac77-418b-bbe2-89c5d1b9f726 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:10:37.878336 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c413c57-ac77-418b-bbe2-89c5d1b9f726 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:10:37.879765 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c413c57-ac77-418b-bbe2-89c5d1b9f726 service neutron] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:10:37.891623 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1348/2696] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Aug 30 02:10:37 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 96 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 02:10:38.395309 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9524708-36ac-4afb-92e0-112cde22172e None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:10:38.396813 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-d9524708-36ac-4afb-92e0-112cde22172e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:10:38.412556 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9524708-36ac-4afb-92e0-112cde22172e None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:10:38.417645 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25cf9689-aea7-4545-ac02-41f4eb601e6b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:10:38.418416 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25cf9689-aea7-4545-ac02-41f4eb601e6b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:10:38.418801 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25cf9689-aea7-4545-ac02-41f4eb601e6b admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:10:38.419220 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25cf9689-aea7-4545-ac02-41f4eb601e6b admin admin] PATH_INFO: `/v3/projects/7efd3cfdc43c4c9aad8ca2e72c723069` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:10:38.420500 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25cf9689-aea7-4545-ac02-41f4eb601e6b admin admin] RBAC: Authorizing `identity:delete_project(project_id=7efd3cfdc43c4c9aad8ca2e72c723069)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:10:38.422707 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25cf9689-aea7-4545-ac02-41f4eb601e6b admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:10:38.536732 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-25cf9689-aea7-4545-ac02-41f4eb601e6b admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '7efd3cfdc43c4c9aad8ca2e72c723069', 'request_id': 'req-25cf9689-aea7-4545-ac02-41f4eb601e6b'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:10:38.541810 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-25cf9689-aea7-4545-ac02-41f4eb601e6b admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '7efd3cfdc43c4c9aad8ca2e72c723069', 'request_id': 'req-25cf9689-aea7-4545-ac02-41f4eb601e6b'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:10:38.549479 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-25cf9689-aea7-4545-ac02-41f4eb601e6b admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '7efd3cfdc43c4c9aad8ca2e72c723069', 'request_id': 'req-25cf9689-aea7-4545-ac02-41f4eb601e6b'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:10:38.870052 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-25cf9689-aea7-4545-ac02-41f4eb601e6b admin admin] The token cache is being invalidate because project 7efd3cfdc43c4c9aad8ca2e72c723069 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81577) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 30 02:10:38.870785 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-25cf9689-aea7-4545-ac02-41f4eb601e6b admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:10:38.871117 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-25cf9689-aea7-4545-ac02-41f4eb601e6b admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:10:38.872700 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1349/2697] 10.222.0.27 () {58 vars in 1233 bytes} [Tue Aug 30 02:10:38 2022] DELETE /identity/v3/projects/7efd3cfdc43c4c9aad8ca2e72c723069 => generated 0 bytes in 478 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 02:10:38.877549 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c413c57-ac77-418b-bbe2-89c5d1b9f726 service neutron] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:10:38.887140 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c413c57-ac77-418b-bbe2-89c5d1b9f726 service neutron] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:10:38.944303 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9500ca9d-01d3-497b-9d21-69a4a4879eda admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:10:38.944960 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9500ca9d-01d3-497b-9d21-69a4a4879eda admin admin] REQUEST_METHOD: `DELETE` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:10:38.945235 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9500ca9d-01d3-497b-9d21-69a4a4879eda admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:10:38.945533 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9500ca9d-01d3-497b-9d21-69a4a4879eda admin admin] PATH_INFO: `/v3/users/bc0c1737e6b648428f06ebc8773f28ea` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:10:38.946696 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9500ca9d-01d3-497b-9d21-69a4a4879eda admin admin] RBAC: Authorizing `identity:delete_user(user_id=bc0c1737e6b648428f06ebc8773f28ea)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:10:38.948682 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9500ca9d-01d3-497b-9d21-69a4a4879eda admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:10:39.184332 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-9500ca9d-01d3-497b-9d21-69a4a4879eda admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'bc0c1737e6b648428f06ebc8773f28ea'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:10:39.185007 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-9500ca9d-01d3-497b-9d21-69a4a4879eda admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'bc0c1737e6b648428f06ebc8773f28ea'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:10:39.253787 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-9500ca9d-01d3-497b-9d21-69a4a4879eda admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'bc0c1737e6b648428f06ebc8773f28ea'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:10:39.254440 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-9500ca9d-01d3-497b-9d21-69a4a4879eda admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'bc0c1737e6b648428f06ebc8773f28ea'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:10:39.265060 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-9500ca9d-01d3-497b-9d21-69a4a4879eda admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'bc0c1737e6b648428f06ebc8773f28ea'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:10:39.287583 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1349/2698] 10.222.0.27 () {58 vars in 1224 bytes} [Tue Aug 30 02:10:38 2022] DELETE /identity/v3/users/bc0c1737e6b648428f06ebc8773f28ea => generated 0 bytes in 411 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 02:10:39.730517 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25cf9689-aea7-4545-ac02-41f4eb601e6b admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:10:39.732007 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-25cf9689-aea7-4545-ac02-41f4eb601e6b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:10:39.754831 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25cf9689-aea7-4545-ac02-41f4eb601e6b admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:10:39.834994 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f503868a-1114-40dc-9a65-fc6c7064fa59 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:10:39.835665 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f503868a-1114-40dc-9a65-fc6c7064fa59 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:10:39.838188 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f503868a-1114-40dc-9a65-fc6c7064fa59 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:10:39.838188 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f503868a-1114-40dc-9a65-fc6c7064fa59 admin admin] PATH_INFO: `/v3/projects/c2b165dfcd5b41818cce106bc8e0c886` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:10:39.838188 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f503868a-1114-40dc-9a65-fc6c7064fa59 admin admin] RBAC: Authorizing `identity:delete_project(project_id=c2b165dfcd5b41818cce106bc8e0c886)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:10:39.839558 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f503868a-1114-40dc-9a65-fc6c7064fa59 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:10:39.843226 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:10:39.950289 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-f503868a-1114-40dc-9a65-fc6c7064fa59 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'c2b165dfcd5b41818cce106bc8e0c886', 'request_id': 'req-f503868a-1114-40dc-9a65-fc6c7064fa59'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:10:39.959320 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-f503868a-1114-40dc-9a65-fc6c7064fa59 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'c2b165dfcd5b41818cce106bc8e0c886', 'request_id': 'req-f503868a-1114-40dc-9a65-fc6c7064fa59'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:10:39.967080 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-f503868a-1114-40dc-9a65-fc6c7064fa59 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'c2b165dfcd5b41818cce106bc8e0c886', 'request_id': 'req-f503868a-1114-40dc-9a65-fc6c7064fa59'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:10:40.030992 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-f503868a-1114-40dc-9a65-fc6c7064fa59 admin admin] The token cache is being invalidate because project c2b165dfcd5b41818cce106bc8e0c886 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81577) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 30 02:10:40.031229 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-f503868a-1114-40dc-9a65-fc6c7064fa59 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:10:40.031924 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-f503868a-1114-40dc-9a65-fc6c7064fa59 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:10:40.034884 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1350/2699] 10.222.0.27 () {58 vars in 1233 bytes} [Tue Aug 30 02:10:39 2022] DELETE /identity/v3/projects/c2b165dfcd5b41818cce106bc8e0c886 => generated 0 bytes in 305 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 02:10:40.039561 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d50c83ae-1ce8-4449-b141-5b9582af242a None None] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:10:40.040077 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d50c83ae-1ce8-4449-b141-5b9582af242a None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:10:40.040178 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d50c83ae-1ce8-4449-b141-5b9582af242a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:10:40.061343 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.auth.core [None req-d50c83ae-1ce8-4449-b141-5b9582af242a None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81578) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:10:40.062605 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-d50c83ae-1ce8-4449-b141-5b9582af242a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:10:40.125035 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:10:40.163143 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1350/2700] 10.222.0.27 () {58 vars in 958 bytes} [Tue Aug 30 02:10:40 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 125 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 02:10:40.250025 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b8816c3-d08e-4c6a-960f-5bce6676333f None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:10:40.250368 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b8816c3-d08e-4c6a-960f-5bce6676333f None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:10:40.251002 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b8816c3-d08e-4c6a-960f-5bce6676333f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:10:40.279791 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-7b8816c3-d08e-4c6a-960f-5bce6676333f None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:10:40.281648 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-7b8816c3-d08e-4c6a-960f-5bce6676333f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:10:40.299461 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1351/2701] 10.222.0.27 () {58 vars in 958 bytes} [Tue Aug 30 02:10:40 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 50 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 02:10:40.300514 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d50c83ae-1ce8-4449-b141-5b9582af242a None None] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:10:40.310879 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d50c83ae-1ce8-4449-b141-5b9582af242a None None] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:10:40.318626 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77fa46e7-76c5-4375-a6b4-4a0dbcb745fe admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:10:40.319543 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77fa46e7-76c5-4375-a6b4-4a0dbcb745fe admin admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:10:40.319998 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77fa46e7-76c5-4375-a6b4-4a0dbcb745fe admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:10:40.320456 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77fa46e7-76c5-4375-a6b4-4a0dbcb745fe admin admin] PATH_INFO: `/v3/domains` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:10:40.321471 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77fa46e7-76c5-4375-a6b4-4a0dbcb745fe admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81578) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 02:10:40.321896 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77fa46e7-76c5-4375-a6b4-4a0dbcb745fe admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:10:40.324264 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77fa46e7-76c5-4375-a6b4-4a0dbcb745fe admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:10:40.339649 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1351/2702] 10.222.0.27 () {58 vars in 1153 bytes} [Tue Aug 30 02:10:40 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 40 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 02:10:40.343980 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b8816c3-d08e-4c6a-960f-5bce6676333f None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:10:40.362005 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b8816c3-d08e-4c6a-960f-5bce6676333f None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:10:40.368572 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad7a39e8-6939-4bc2-84c9-a46696c18a95 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:10:40.369290 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad7a39e8-6939-4bc2-84c9-a46696c18a95 admin admin] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:10:40.369449 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad7a39e8-6939-4bc2-84c9-a46696c18a95 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:10:40.369754 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad7a39e8-6939-4bc2-84c9-a46696c18a95 admin admin] PATH_INFO: `/v3/projects` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:10:40.370409 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad7a39e8-6939-4bc2-84c9-a46696c18a95 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:10:40.372701 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad7a39e8-6939-4bc2-84c9-a46696c18a95 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:10:40.478512 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1352/2703] 10.222.0.27 () {60 vars in 1153 bytes} [Tue Aug 30 02:10:40 2022] POST /identity/v3/projects => generated 375 bytes in 135 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 02:10:40.482808 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77fa46e7-76c5-4375-a6b4-4a0dbcb745fe admin admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:10:40.501109 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77fa46e7-76c5-4375-a6b4-4a0dbcb745fe admin admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:10:40.568415 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2639741-bd9b-4ada-88cf-4737ecbc040b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:10:40.569407 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2639741-bd9b-4ada-88cf-4737ecbc040b admin admin] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:10:40.569798 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2639741-bd9b-4ada-88cf-4737ecbc040b admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:10:40.570069 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2639741-bd9b-4ada-88cf-4737ecbc040b admin admin] PATH_INFO: `/v3/users` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:10:40.570891 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2639741-bd9b-4ada-88cf-4737ecbc040b admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:10:40.572788 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2639741-bd9b-4ada-88cf-4737ecbc040b admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:10:40.688130 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1352/2704] 10.222.0.27 () {60 vars in 1144 bytes} [Tue Aug 30 02:10:40 2022] POST /identity/v3/users => generated 288 bytes in 206 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 02:10:40.693388 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad7a39e8-6939-4bc2-84c9-a46696c18a95 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:10:40.712563 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad7a39e8-6939-4bc2-84c9-a46696c18a95 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:10:40.724595 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd90661b-474a-4fec-b456-f0059e398932 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:10:40.725339 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd90661b-474a-4fec-b456-f0059e398932 admin admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:10:40.725736 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd90661b-474a-4fec-b456-f0059e398932 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:10:40.726075 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd90661b-474a-4fec-b456-f0059e398932 admin admin] PATH_INFO: `/v3/roles` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:10:40.726832 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd90661b-474a-4fec-b456-f0059e398932 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:10:40.728891 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd90661b-474a-4fec-b456-f0059e398932 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:10:40.742940 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1353/2705] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Aug 30 02:10:40 2022] GET /identity/v3/roles => generated 1420 bytes in 50 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 02:10:40.746930 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2639741-bd9b-4ada-88cf-4737ecbc040b admin admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:10:40.759289 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2639741-bd9b-4ada-88cf-4737ecbc040b admin admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:10:40.764191 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-705c0e70-ac14-43fe-9ccc-9646af4e6c72 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:10:40.764798 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-705c0e70-ac14-43fe-9ccc-9646af4e6c72 admin admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:10:40.765089 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-705c0e70-ac14-43fe-9ccc-9646af4e6c72 admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:10:40.765548 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-705c0e70-ac14-43fe-9ccc-9646af4e6c72 admin admin] PATH_INFO: `/v3/roles` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:10:40.766124 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-705c0e70-ac14-43fe-9ccc-9646af4e6c72 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:10:40.767979 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-705c0e70-ac14-43fe-9ccc-9646af4e6c72 admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:10:40.780237 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1353/2706] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Aug 30 02:10:40 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 30 02:10:40.785071 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd90661b-474a-4fec-b456-f0059e398932 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:10:40.797481 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd90661b-474a-4fec-b456-f0059e398932 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:10:40.804459 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2d17206-6886-4a72-8b21-93d60b048efe admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:10:40.805122 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2d17206-6886-4a72-8b21-93d60b048efe admin admin] REQUEST_METHOD: `PUT` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:10:40.805409 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2d17206-6886-4a72-8b21-93d60b048efe admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:10:40.805717 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2d17206-6886-4a72-8b21-93d60b048efe admin admin] PATH_INFO: `/v3/projects/1ff795e7346e468cb175ae7647589c32/users/01b6c5532e8e4e6daa37c00ae8504e71/roles/1fb81afcb9f245f3b989add94598a063` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:10:40.834323 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2d17206-6886-4a72-8b21-93d60b048efe admin admin] RBAC: Authorizing `identity:create_grant(project_id=1ff795e7346e468cb175ae7647589c32, user_id=01b6c5532e8e4e6daa37c00ae8504e71, role_id=1fb81afcb9f245f3b989add94598a063)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:10:40.838471 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2d17206-6886-4a72-8b21-93d60b048efe admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:10:40.909288 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1354/2707] 10.222.0.27 () {60 vars in 1483 bytes} [Tue Aug 30 02:10:40 2022] PUT /identity/v3/projects/1ff795e7346e468cb175ae7647589c32/users/01b6c5532e8e4e6daa37c00ae8504e71/roles/1fb81afcb9f245f3b989add94598a063 => generated 0 bytes in 125 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 30 02:10:40.925136 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47fa7a58-7732-40db-bc24-5bc3cc6c886c None None] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:10:40.925558 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47fa7a58-7732-40db-bc24-5bc3cc6c886c None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:10:40.925711 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47fa7a58-7732-40db-bc24-5bc3cc6c886c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:10:40.962114 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.auth.core [None req-47fa7a58-7732-40db-bc24-5bc3cc6c886c None None] MFA Rules not processed for user `01b6c5532e8e4e6daa37c00ae8504e71`. Rule list: `[]` (Enabled: `True`). {{(pid=81578) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:10:40.964364 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-47fa7a58-7732-40db-bc24-5bc3cc6c886c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:10:41.054589 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1354/2708] 10.222.0.27 () {58 vars in 958 bytes} [Tue Aug 30 02:10:40 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 131 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 02:10:41.070684 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2d17206-6886-4a72-8b21-93d60b048efe admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:10:41.073619 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-d2d17206-6886-4a72-8b21-93d60b048efe admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:10:41.154254 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2d17206-6886-4a72-8b21-93d60b048efe admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:10:41.160495 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65745588-4309-425d-ac03-32f94cf29799 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ab936b3767a8417aa80c8c8f1b84b667', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4562f3951fbe465280073e60a11e44e2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:10:41.161325 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65745588-4309-425d-ac03-32f94cf29799 service cinder] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:10:41.161678 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65745588-4309-425d-ac03-32f94cf29799 service cinder] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:10:41.162036 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65745588-4309-425d-ac03-32f94cf29799 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:10:41.174595 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65745588-4309-425d-ac03-32f94cf29799 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:10:41.176173 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65745588-4309-425d-ac03-32f94cf29799 service cinder] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:10:41.186724 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1355/2709] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Aug 30 02:10:41 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 116 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 30 02:10:45.554848 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47fa7a58-7732-40db-bc24-5bc3cc6c886c None None] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:10:45.604121 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:10:45.624406 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47fa7a58-7732-40db-bc24-5bc3cc6c886c None None] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:10:45.629558 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b62e98f-d3a9-4a89-bc63-9355bcb49eb5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:10:45.630199 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b62e98f-d3a9-4a89-bc63-9355bcb49eb5 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:10:45.630477 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b62e98f-d3a9-4a89-bc63-9355bcb49eb5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:10:45.630748 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b62e98f-d3a9-4a89-bc63-9355bcb49eb5 admin admin] PATH_INFO: `/v3/users/01b6c5532e8e4e6daa37c00ae8504e71` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:10:45.631937 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b62e98f-d3a9-4a89-bc63-9355bcb49eb5 admin admin] RBAC: Authorizing `identity:delete_user(user_id=01b6c5532e8e4e6daa37c00ae8504e71)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:10:45.633855 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b62e98f-d3a9-4a89-bc63-9355bcb49eb5 admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:10:45.846498 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-0b62e98f-d3a9-4a89-bc63-9355bcb49eb5 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '01b6c5532e8e4e6daa37c00ae8504e71'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:10:45.847239 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-0b62e98f-d3a9-4a89-bc63-9355bcb49eb5 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '01b6c5532e8e4e6daa37c00ae8504e71'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:10:45.915239 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-0b62e98f-d3a9-4a89-bc63-9355bcb49eb5 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '01b6c5532e8e4e6daa37c00ae8504e71'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:10:45.915823 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-0b62e98f-d3a9-4a89-bc63-9355bcb49eb5 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '01b6c5532e8e4e6daa37c00ae8504e71'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:10:45.923393 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-0b62e98f-d3a9-4a89-bc63-9355bcb49eb5 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '01b6c5532e8e4e6daa37c00ae8504e71'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:10:45.940601 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1355/2710] 10.222.0.27 () {58 vars in 1224 bytes} [Tue Aug 30 02:10:45 2022] DELETE /identity/v3/users/01b6c5532e8e4e6daa37c00ae8504e71 => generated 0 bytes in 386 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 02:10:45.945164 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c54f65f0-1e38-4ac9-a641-2a4734295c28 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:10:45.945739 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c54f65f0-1e38-4ac9-a641-2a4734295c28 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:10:45.945841 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c54f65f0-1e38-4ac9-a641-2a4734295c28 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:10:45.965340 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-c54f65f0-1e38-4ac9-a641-2a4734295c28 None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:10:45.966568 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-c54f65f0-1e38-4ac9-a641-2a4734295c28 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:10:45.992824 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:10:46.026370 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1356/2711] 10.222.0.27 () {58 vars in 958 bytes} [Tue Aug 30 02:10:45 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 82 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 02:10:46.036240 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b62e98f-d3a9-4a89-bc63-9355bcb49eb5 admin admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:10:46.037664 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-0b62e98f-d3a9-4a89-bc63-9355bcb49eb5 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:10:46.098026 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b62e98f-d3a9-4a89-bc63-9355bcb49eb5 admin admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:10:46.101667 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c01bb675-b4f9-4531-9be7-9b33c623fcc5 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f3ea1580c56f4814a92bbb2acf542905', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4562f3951fbe465280073e60a11e44e2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:10:46.102265 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c01bb675-b4f9-4531-9be7-9b33c623fcc5 service neutron] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:10:46.102546 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c01bb675-b4f9-4531-9be7-9b33c623fcc5 service neutron] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:10:46.102816 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c01bb675-b4f9-4531-9be7-9b33c623fcc5 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:10:46.112821 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c01bb675-b4f9-4531-9be7-9b33c623fcc5 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:10:46.114473 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c01bb675-b4f9-4531-9be7-9b33c623fcc5 service neutron] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:10:46.125442 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1356/2712] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Aug 30 02:10:46 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 90 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 02:10:46.502280 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-066f2646-f8e5-48ba-878d-a6a07b03c429 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:10:46.502569 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-066f2646-f8e5-48ba-878d-a6a07b03c429 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:10:46.502839 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-066f2646-f8e5-48ba-878d-a6a07b03c429 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:10:46.521967 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-066f2646-f8e5-48ba-878d-a6a07b03c429 None None] MFA Rules not processed for user `121c374d49a64335ab74a431cfbc1320`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:10:46.523200 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-066f2646-f8e5-48ba-878d-a6a07b03c429 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:10:46.571123 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c01bb675-b4f9-4531-9be7-9b33c623fcc5 service neutron] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:10:46.573548 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-c01bb675-b4f9-4531-9be7-9b33c623fcc5 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:10:46.590057 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c01bb675-b4f9-4531-9be7-9b33c623fcc5 service neutron] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:10:46.594373 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1357/2713] 10.222.0.27 () {58 vars in 967 bytes} [Tue Aug 30 02:10:46 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 02:10:46.595627 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed8f21ad-c579-4d1a-80a5-98085a90b135 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:10:46.596343 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed8f21ad-c579-4d1a-80a5-98085a90b135 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:10:46.596785 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed8f21ad-c579-4d1a-80a5-98085a90b135 admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:10:46.597175 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed8f21ad-c579-4d1a-80a5-98085a90b135 admin admin] PATH_INFO: `/v3/projects/1ff795e7346e468cb175ae7647589c32` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:10:46.598534 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed8f21ad-c579-4d1a-80a5-98085a90b135 admin admin] RBAC: Authorizing `identity:delete_project(project_id=1ff795e7346e468cb175ae7647589c32)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:10:46.601877 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed8f21ad-c579-4d1a-80a5-98085a90b135 admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:10:46.609089 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-066f2646-f8e5-48ba-878d-a6a07b03c429 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:10:46.611145 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-066f2646-f8e5-48ba-878d-a6a07b03c429 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:10:46.692345 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-066f2646-f8e5-48ba-878d-a6a07b03c429 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:10:46.697255 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f18ecf7-533f-4719-9d1f-1e5b81821fd0 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b4f0008fde424f22bc1f7307cf422774', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4562f3951fbe465280073e60a11e44e2', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:10:46.697900 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f18ecf7-533f-4719-9d1f-1e5b81821fd0 service swift] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:10:46.698180 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f18ecf7-533f-4719-9d1f-1e5b81821fd0 service swift] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:10:46.698449 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f18ecf7-533f-4719-9d1f-1e5b81821fd0 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:10:46.707182 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f18ecf7-533f-4719-9d1f-1e5b81821fd0 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:10:46.708951 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f18ecf7-533f-4719-9d1f-1e5b81821fd0 service swift] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:10:46.717343 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1358/2714] 10.222.0.27 () {60 vars in 1384 bytes} [Tue Aug 30 02:10:46 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 109 msecs (HTTP/1.1 200) 6 headers in 379 bytes (2 switches on core 0) Aug 30 02:10:46.760159 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-ed8f21ad-c579-4d1a-80a5-98085a90b135 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '1ff795e7346e468cb175ae7647589c32', 'request_id': 'req-ed8f21ad-c579-4d1a-80a5-98085a90b135'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:10:46.765699 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-ed8f21ad-c579-4d1a-80a5-98085a90b135 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '1ff795e7346e468cb175ae7647589c32', 'request_id': 'req-ed8f21ad-c579-4d1a-80a5-98085a90b135'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:10:46.766873 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=83,keystone:DELETE=16,keystone:INSERT=1 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:10:46.772556 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-ed8f21ad-c579-4d1a-80a5-98085a90b135 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '1ff795e7346e468cb175ae7647589c32', 'request_id': 'req-ed8f21ad-c579-4d1a-80a5-98085a90b135'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:10:46.844443 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-ed8f21ad-c579-4d1a-80a5-98085a90b135 admin admin] The token cache is being invalidate because project 1ff795e7346e468cb175ae7647589c32 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81578) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 30 02:10:46.844736 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-ed8f21ad-c579-4d1a-80a5-98085a90b135 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:10:46.845978 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-ed8f21ad-c579-4d1a-80a5-98085a90b135 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:10:46.847468 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1357/2715] 10.222.0.27 () {58 vars in 1233 bytes} [Tue Aug 30 02:10:46 2022] DELETE /identity/v3/projects/1ff795e7346e468cb175ae7647589c32 => generated 0 bytes in 277 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 02:10:46.854181 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d17713df-15cc-4009-8337-13ad988dfffa None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:10:46.854694 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d17713df-15cc-4009-8337-13ad988dfffa None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:10:46.856200 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d17713df-15cc-4009-8337-13ad988dfffa None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:10:46.877836 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-d17713df-15cc-4009-8337-13ad988dfffa None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:10:46.879051 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-d17713df-15cc-4009-8337-13ad988dfffa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:10:46.939575 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1359/2716] 10.222.0.27 () {58 vars in 958 bytes} [Tue Aug 30 02:10:46 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 02:10:46.947615 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd5b613a-3a48-4c54-9444-3cf8bd0b512b None None] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:10:46.948029 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd5b613a-3a48-4c54-9444-3cf8bd0b512b None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:10:46.948969 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd5b613a-3a48-4c54-9444-3cf8bd0b512b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:10:46.968952 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.auth.core [None req-fd5b613a-3a48-4c54-9444-3cf8bd0b512b None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81578) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:10:46.970806 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-fd5b613a-3a48-4c54-9444-3cf8bd0b512b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:10:46.978636 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1358/2717] 10.222.0.27 () {58 vars in 958 bytes} [Tue Aug 30 02:10:46 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 30 02:10:46.983160 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d17713df-15cc-4009-8337-13ad988dfffa None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:10:46.994506 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d17713df-15cc-4009-8337-13ad988dfffa None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:10:46.999052 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64365358-7ebc-4631-bcc6-787c5e57189c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:10:46.999656 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64365358-7ebc-4631-bcc6-787c5e57189c admin admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:10:46.999923 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64365358-7ebc-4631-bcc6-787c5e57189c admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:10:47.000186 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64365358-7ebc-4631-bcc6-787c5e57189c admin admin] PATH_INFO: `/v3/domains` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:10:47.000823 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64365358-7ebc-4631-bcc6-787c5e57189c admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 02:10:47.001122 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64365358-7ebc-4631-bcc6-787c5e57189c admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:10:47.002965 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64365358-7ebc-4631-bcc6-787c5e57189c admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:10:47.004472 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:10:47.014776 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1360/2718] 10.222.0.27 () {58 vars in 1153 bytes} [Tue Aug 30 02:10:46 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 02:10:47.019261 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd5b613a-3a48-4c54-9444-3cf8bd0b512b None None] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:10:47.031489 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd5b613a-3a48-4c54-9444-3cf8bd0b512b None None] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:10:47.035339 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf54dfc0-0fc4-4065-97d5-0869f368f52a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:10:47.035943 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf54dfc0-0fc4-4065-97d5-0869f368f52a admin admin] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:10:47.036220 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf54dfc0-0fc4-4065-97d5-0869f368f52a admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:10:47.036486 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf54dfc0-0fc4-4065-97d5-0869f368f52a admin admin] PATH_INFO: `/v3/projects` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:10:47.037922 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf54dfc0-0fc4-4065-97d5-0869f368f52a admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:10:47.040069 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf54dfc0-0fc4-4065-97d5-0869f368f52a admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:10:47.168705 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1359/2719] 10.222.0.27 () {60 vars in 1153 bytes} [Tue Aug 30 02:10:47 2022] POST /identity/v3/projects => generated 375 bytes in 150 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 02:10:47.173236 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64365358-7ebc-4631-bcc6-787c5e57189c admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:10:47.183253 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64365358-7ebc-4631-bcc6-787c5e57189c admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:10:47.242551 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f56ab996-f27f-433a-9da5-9f4c6f338f4d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:10:47.243148 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f56ab996-f27f-433a-9da5-9f4c6f338f4d admin admin] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:10:47.243432 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f56ab996-f27f-433a-9da5-9f4c6f338f4d admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:10:47.243705 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f56ab996-f27f-433a-9da5-9f4c6f338f4d admin admin] PATH_INFO: `/v3/users` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:10:47.244412 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f56ab996-f27f-433a-9da5-9f4c6f338f4d admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:10:47.246452 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f56ab996-f27f-433a-9da5-9f4c6f338f4d admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:10:47.330528 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1361/2720] 10.222.0.27 () {60 vars in 1144 bytes} [Tue Aug 30 02:10:47 2022] POST /identity/v3/users => generated 288 bytes in 156 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 02:10:47.333416 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf54dfc0-0fc4-4065-97d5-0869f368f52a admin admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:10:47.343999 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf54dfc0-0fc4-4065-97d5-0869f368f52a admin admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:10:47.349538 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0dc926ae-32d0-40ce-a3bb-2febbc4dad68 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:10:47.350199 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dc926ae-32d0-40ce-a3bb-2febbc4dad68 admin admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:10:47.350485 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dc926ae-32d0-40ce-a3bb-2febbc4dad68 admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:10:47.350758 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dc926ae-32d0-40ce-a3bb-2febbc4dad68 admin admin] PATH_INFO: `/v3/roles` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:10:47.351402 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0dc926ae-32d0-40ce-a3bb-2febbc4dad68 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:10:47.352980 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0dc926ae-32d0-40ce-a3bb-2febbc4dad68 admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:10:47.363560 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1360/2721] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Aug 30 02:10:47 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 02:10:47.367889 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f56ab996-f27f-433a-9da5-9f4c6f338f4d admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:10:47.378469 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f56ab996-f27f-433a-9da5-9f4c6f338f4d admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:10:47.383396 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad9fcbc5-d11d-46d8-bf01-393c05adb7d7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:10:47.384000 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad9fcbc5-d11d-46d8-bf01-393c05adb7d7 admin admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:10:47.384999 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad9fcbc5-d11d-46d8-bf01-393c05adb7d7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:10:47.385274 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad9fcbc5-d11d-46d8-bf01-393c05adb7d7 admin admin] PATH_INFO: `/v3/roles` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:10:47.385954 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad9fcbc5-d11d-46d8-bf01-393c05adb7d7 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:10:47.387539 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad9fcbc5-d11d-46d8-bf01-393c05adb7d7 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:10:47.400231 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1362/2722] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Aug 30 02:10:47 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 02:10:47.404408 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0dc926ae-32d0-40ce-a3bb-2febbc4dad68 admin admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:10:47.415128 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0dc926ae-32d0-40ce-a3bb-2febbc4dad68 admin admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:10:47.419796 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a19b59fb-4b91-4e90-9259-31580a102158 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:10:47.420451 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a19b59fb-4b91-4e90-9259-31580a102158 admin admin] REQUEST_METHOD: `PUT` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:10:47.420715 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a19b59fb-4b91-4e90-9259-31580a102158 admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:10:47.420983 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a19b59fb-4b91-4e90-9259-31580a102158 admin admin] PATH_INFO: `/v3/projects/c25441166f7246b4b999cdd7383ad358/users/8922516b033d4e24b2ec8d0deb432267/roles/1fb81afcb9f245f3b989add94598a063` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:10:47.436403 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a19b59fb-4b91-4e90-9259-31580a102158 admin admin] RBAC: Authorizing `identity:create_grant(project_id=c25441166f7246b4b999cdd7383ad358, user_id=8922516b033d4e24b2ec8d0deb432267, role_id=1fb81afcb9f245f3b989add94598a063)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:10:47.439139 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a19b59fb-4b91-4e90-9259-31580a102158 admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:10:47.526161 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1361/2723] 10.222.0.27 () {60 vars in 1483 bytes} [Tue Aug 30 02:10:47 2022] PUT /identity/v3/projects/c25441166f7246b4b999cdd7383ad358/users/8922516b033d4e24b2ec8d0deb432267/roles/1fb81afcb9f245f3b989add94598a063 => generated 0 bytes in 122 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 02:10:47.538701 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c674df7a-1f84-40c4-92aa-90cac2c655ae None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:10:47.539106 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c674df7a-1f84-40c4-92aa-90cac2c655ae None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:10:47.546576 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c674df7a-1f84-40c4-92aa-90cac2c655ae None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:10:47.579067 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-c674df7a-1f84-40c4-92aa-90cac2c655ae None None] MFA Rules not processed for user `8922516b033d4e24b2ec8d0deb432267`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:10:47.580366 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-c674df7a-1f84-40c4-92aa-90cac2c655ae None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:10:47.663586 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1363/2724] 10.222.0.27 () {58 vars in 958 bytes} [Tue Aug 30 02:10:47 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 126 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 02:10:47.668483 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a19b59fb-4b91-4e90-9259-31580a102158 admin admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:10:47.745194 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a19b59fb-4b91-4e90-9259-31580a102158 admin admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:10:47.749829 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc808bc4-c7e3-444c-81b3-ee05f68055e4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:10:47.750438 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc808bc4-c7e3-444c-81b3-ee05f68055e4 admin admin] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:10:47.750720 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc808bc4-c7e3-444c-81b3-ee05f68055e4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:10:47.750989 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc808bc4-c7e3-444c-81b3-ee05f68055e4 admin admin] PATH_INFO: `/v3/projects` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:10:47.751647 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc808bc4-c7e3-444c-81b3-ee05f68055e4 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:10:47.753549 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc808bc4-c7e3-444c-81b3-ee05f68055e4 admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:10:47.866951 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1362/2725] 10.222.0.27 () {60 vars in 1153 bytes} [Tue Aug 30 02:10:47 2022] POST /identity/v3/projects => generated 375 bytes in 199 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 02:10:47.871037 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c674df7a-1f84-40c4-92aa-90cac2c655ae None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:10:47.881916 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c674df7a-1f84-40c4-92aa-90cac2c655ae None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:10:47.937885 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb3157ea-8837-4f7c-9c64-b26341fcd673 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:10:47.938844 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb3157ea-8837-4f7c-9c64-b26341fcd673 admin admin] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:10:47.939163 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb3157ea-8837-4f7c-9c64-b26341fcd673 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:10:47.939500 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb3157ea-8837-4f7c-9c64-b26341fcd673 admin admin] PATH_INFO: `/v3/users` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:10:47.940154 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb3157ea-8837-4f7c-9c64-b26341fcd673 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:10:47.942073 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb3157ea-8837-4f7c-9c64-b26341fcd673 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:10:47.951103 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:10:48.037877 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1364/2726] 10.222.0.27 () {60 vars in 1144 bytes} [Tue Aug 30 02:10:47 2022] POST /identity/v3/users => generated 288 bytes in 167 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 02:10:48.043203 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc808bc4-c7e3-444c-81b3-ee05f68055e4 admin admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:10:48.056329 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc808bc4-c7e3-444c-81b3-ee05f68055e4 admin admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:10:48.066979 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a69c2bc-687f-47be-bddb-8ae45c4b057f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:10:48.070724 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a69c2bc-687f-47be-bddb-8ae45c4b057f admin admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:10:48.070724 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a69c2bc-687f-47be-bddb-8ae45c4b057f admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:10:48.070724 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a69c2bc-687f-47be-bddb-8ae45c4b057f admin admin] PATH_INFO: `/v3/roles` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:10:48.070724 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a69c2bc-687f-47be-bddb-8ae45c4b057f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:10:48.070724 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a69c2bc-687f-47be-bddb-8ae45c4b057f admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:10:48.082243 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1363/2727] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Aug 30 02:10:48 2022] GET /identity/v3/roles => generated 1420 bytes in 40 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 02:10:48.086231 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb3157ea-8837-4f7c-9c64-b26341fcd673 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:10:48.097394 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb3157ea-8837-4f7c-9c64-b26341fcd673 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:10:48.102205 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2a4f3c2-c968-475b-acfc-05d8535aae6f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:10:48.102795 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2a4f3c2-c968-475b-acfc-05d8535aae6f admin admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:10:48.103074 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2a4f3c2-c968-475b-acfc-05d8535aae6f admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:10:48.103367 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2a4f3c2-c968-475b-acfc-05d8535aae6f admin admin] PATH_INFO: `/v3/roles` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:10:48.104007 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2a4f3c2-c968-475b-acfc-05d8535aae6f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:10:48.105901 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2a4f3c2-c968-475b-acfc-05d8535aae6f admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:10:48.116497 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1365/2728] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Aug 30 02:10:48 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 02:10:48.120843 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a69c2bc-687f-47be-bddb-8ae45c4b057f admin admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:10:48.131975 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a69c2bc-687f-47be-bddb-8ae45c4b057f admin admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:10:48.140955 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79ee1973-5ee6-46b3-aee0-03ab61cab27d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:10:48.141662 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79ee1973-5ee6-46b3-aee0-03ab61cab27d admin admin] REQUEST_METHOD: `PUT` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:10:48.142623 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79ee1973-5ee6-46b3-aee0-03ab61cab27d admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:10:48.143086 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79ee1973-5ee6-46b3-aee0-03ab61cab27d admin admin] PATH_INFO: `/v3/projects/ee3fa3bd0a1d471d9fddc60930e62bb3/users/7e93952263124221b28466603da9aa65/roles/1fb81afcb9f245f3b989add94598a063` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:10:48.159788 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79ee1973-5ee6-46b3-aee0-03ab61cab27d admin admin] RBAC: Authorizing `identity:create_grant(project_id=ee3fa3bd0a1d471d9fddc60930e62bb3, user_id=7e93952263124221b28466603da9aa65, role_id=1fb81afcb9f245f3b989add94598a063)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:10:48.162643 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79ee1973-5ee6-46b3-aee0-03ab61cab27d admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:10:48.250842 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1364/2729] 10.222.0.27 () {60 vars in 1483 bytes} [Tue Aug 30 02:10:48 2022] PUT /identity/v3/projects/ee3fa3bd0a1d471d9fddc60930e62bb3/users/7e93952263124221b28466603da9aa65/roles/1fb81afcb9f245f3b989add94598a063 => generated 0 bytes in 131 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 30 02:10:48.261949 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-658a6866-a703-4f2d-83e8-841ed61d8b91 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:10:48.262486 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-658a6866-a703-4f2d-83e8-841ed61d8b91 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:10:48.262856 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-658a6866-a703-4f2d-83e8-841ed61d8b91 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:10:48.291790 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-658a6866-a703-4f2d-83e8-841ed61d8b91 None None] MFA Rules not processed for user `7e93952263124221b28466603da9aa65`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:10:48.293022 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-658a6866-a703-4f2d-83e8-841ed61d8b91 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:10:48.359107 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1366/2730] 10.222.0.27 () {58 vars in 958 bytes} [Tue Aug 30 02:10:48 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 02:10:48.364390 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79ee1973-5ee6-46b3-aee0-03ab61cab27d admin admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:10:48.400808 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=94,keystone:INSERT=5,keystone:DELETE=1 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:10:48.426976 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79ee1973-5ee6-46b3-aee0-03ab61cab27d admin admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:10:48.431727 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-283dce9d-b732-46c4-a248-6d5c80d05c12 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:10:48.432330 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-283dce9d-b732-46c4-a248-6d5c80d05c12 admin admin] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:10:48.436749 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-283dce9d-b732-46c4-a248-6d5c80d05c12 admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:10:48.436749 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-283dce9d-b732-46c4-a248-6d5c80d05c12 admin admin] PATH_INFO: `/v3/projects` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:10:48.436749 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-283dce9d-b732-46c4-a248-6d5c80d05c12 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:10:48.436749 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-283dce9d-b732-46c4-a248-6d5c80d05c12 admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:10:48.559462 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1365/2731] 10.222.0.27 () {60 vars in 1153 bytes} [Tue Aug 30 02:10:48 2022] POST /identity/v3/projects => generated 375 bytes in 196 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 02:10:48.563655 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-658a6866-a703-4f2d-83e8-841ed61d8b91 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:10:48.574851 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-658a6866-a703-4f2d-83e8-841ed61d8b91 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:10:48.631769 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74c7efcb-0839-46e6-8743-98ad712b156d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:10:48.632376 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74c7efcb-0839-46e6-8743-98ad712b156d admin admin] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:10:48.633164 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74c7efcb-0839-46e6-8743-98ad712b156d admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:10:48.633164 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74c7efcb-0839-46e6-8743-98ad712b156d admin admin] PATH_INFO: `/v3/users` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:10:48.633570 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74c7efcb-0839-46e6-8743-98ad712b156d admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:10:48.635475 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74c7efcb-0839-46e6-8743-98ad712b156d admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:10:48.713980 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1367/2732] 10.222.0.27 () {60 vars in 1144 bytes} [Tue Aug 30 02:10:48 2022] POST /identity/v3/users => generated 294 bytes in 151 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 02:10:48.718133 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-283dce9d-b732-46c4-a248-6d5c80d05c12 admin admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:10:48.729396 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-283dce9d-b732-46c4-a248-6d5c80d05c12 admin admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:10:48.734370 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05c667d2-1130-4e34-a00b-131d05711906 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:10:48.734962 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05c667d2-1130-4e34-a00b-131d05711906 admin admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:10:48.735238 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05c667d2-1130-4e34-a00b-131d05711906 admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:10:48.735506 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05c667d2-1130-4e34-a00b-131d05711906 admin admin] PATH_INFO: `/v3/roles` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:10:48.736145 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05c667d2-1130-4e34-a00b-131d05711906 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:10:48.737730 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05c667d2-1130-4e34-a00b-131d05711906 admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:10:48.748133 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1366/2733] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Aug 30 02:10:48 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 02:10:48.752566 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74c7efcb-0839-46e6-8743-98ad712b156d admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:10:48.762489 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74c7efcb-0839-46e6-8743-98ad712b156d admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:10:48.767674 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1382b8e-4b2d-4106-a816-69d86336d34a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:10:48.768324 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1382b8e-4b2d-4106-a816-69d86336d34a admin admin] REQUEST_METHOD: `PUT` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:10:48.768608 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1382b8e-4b2d-4106-a816-69d86336d34a admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:10:48.768939 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1382b8e-4b2d-4106-a816-69d86336d34a admin admin] PATH_INFO: `/v3/projects/733a23ea0e3945d9ab688b994ebe1192/users/0ed9ee2708c6439a9bd2401310a54cbc/roles/ead5fac86a0747ea8823a82efcea8ed7` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:10:48.784254 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1382b8e-4b2d-4106-a816-69d86336d34a admin admin] RBAC: Authorizing `identity:create_grant(project_id=733a23ea0e3945d9ab688b994ebe1192, user_id=0ed9ee2708c6439a9bd2401310a54cbc, role_id=ead5fac86a0747ea8823a82efcea8ed7)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:10:48.787209 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1382b8e-4b2d-4106-a816-69d86336d34a admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:10:48.861924 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1368/2734] 10.222.0.27 () {60 vars in 1483 bytes} [Tue Aug 30 02:10:48 2022] PUT /identity/v3/projects/733a23ea0e3945d9ab688b994ebe1192/users/0ed9ee2708c6439a9bd2401310a54cbc/roles/ead5fac86a0747ea8823a82efcea8ed7 => generated 0 bytes in 110 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 30 02:10:48.866623 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05c667d2-1130-4e34-a00b-131d05711906 admin admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:10:48.878633 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05c667d2-1130-4e34-a00b-131d05711906 admin admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:10:48.934101 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61456625-1c9b-400e-b19b-ae86d8cee7da admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:10:48.934701 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61456625-1c9b-400e-b19b-ae86d8cee7da admin admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:10:48.934982 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61456625-1c9b-400e-b19b-ae86d8cee7da admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:10:48.935253 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61456625-1c9b-400e-b19b-ae86d8cee7da admin admin] PATH_INFO: `/v3/roles` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:10:48.935884 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61456625-1c9b-400e-b19b-ae86d8cee7da admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:10:48.937450 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61456625-1c9b-400e-b19b-ae86d8cee7da admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:10:48.946981 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1367/2735] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Aug 30 02:10:48 2022] GET /identity/v3/roles => generated 1420 bytes in 82 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 02:10:48.951241 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1382b8e-4b2d-4106-a816-69d86336d34a admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:10:48.962100 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1382b8e-4b2d-4106-a816-69d86336d34a admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:10:48.966620 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc6d48d0-6f7d-44d8-85d2-a9adf54f7dfc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:10:48.967257 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc6d48d0-6f7d-44d8-85d2-a9adf54f7dfc admin admin] REQUEST_METHOD: `PUT` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:10:48.967533 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc6d48d0-6f7d-44d8-85d2-a9adf54f7dfc admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:10:48.967800 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc6d48d0-6f7d-44d8-85d2-a9adf54f7dfc admin admin] PATH_INFO: `/v3/projects/733a23ea0e3945d9ab688b994ebe1192/users/0ed9ee2708c6439a9bd2401310a54cbc/roles/ead5fac86a0747ea8823a82efcea8ed7` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:10:48.969765 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc6d48d0-6f7d-44d8-85d2-a9adf54f7dfc admin admin] RBAC: Authorizing `identity:create_grant(project_id=733a23ea0e3945d9ab688b994ebe1192, user_id=0ed9ee2708c6439a9bd2401310a54cbc, role_id=ead5fac86a0747ea8823a82efcea8ed7)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:10:48.973085 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc6d48d0-6f7d-44d8-85d2-a9adf54f7dfc admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:10:48.983300 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1369/2736] 10.222.0.27 () {60 vars in 1483 bytes} [Tue Aug 30 02:10:48 2022] PUT /identity/v3/projects/733a23ea0e3945d9ab688b994ebe1192/users/0ed9ee2708c6439a9bd2401310a54cbc/roles/ead5fac86a0747ea8823a82efcea8ed7 => generated 0 bytes in 33 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 30 02:10:48.994695 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07ef4305-6dc3-479b-b3ed-c4e4d36002c1 None None] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:10:48.994979 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07ef4305-6dc3-479b-b3ed-c4e4d36002c1 None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:10:48.995251 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07ef4305-6dc3-479b-b3ed-c4e4d36002c1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:10:49.024112 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.auth.core [None req-07ef4305-6dc3-479b-b3ed-c4e4d36002c1 None None] MFA Rules not processed for user `0ed9ee2708c6439a9bd2401310a54cbc`. Rule list: `[]` (Enabled: `True`). {{(pid=81578) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:10:49.025317 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-07ef4305-6dc3-479b-b3ed-c4e4d36002c1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:10:49.097681 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1368/2737] 10.222.0.27 () {58 vars in 958 bytes} [Tue Aug 30 02:10:48 2022] POST /identity/v3/auth/tokens => generated 3319 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 02:10:49.116353 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc6d48d0-6f7d-44d8-85d2-a9adf54f7dfc admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:10:49.118135 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-cc6d48d0-6f7d-44d8-85d2-a9adf54f7dfc admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:10:49.156817 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:INSERT=7,keystone:SELECT=93 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:10:49.182417 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc6d48d0-6f7d-44d8-85d2-a9adf54f7dfc admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:10:49.186564 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cea44e9-d1f6-432a-81c0-377c11c8a4ae service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ab936b3767a8417aa80c8c8f1b84b667', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4562f3951fbe465280073e60a11e44e2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:10:49.187167 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cea44e9-d1f6-432a-81c0-377c11c8a4ae service cinder] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:10:49.187444 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cea44e9-d1f6-432a-81c0-377c11c8a4ae service cinder] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:10:49.187713 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cea44e9-d1f6-432a-81c0-377c11c8a4ae service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:10:49.250070 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7cea44e9-d1f6-432a-81c0-377c11c8a4ae service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:10:49.252363 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7cea44e9-d1f6-432a-81c0-377c11c8a4ae service cinder] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:10:49.261498 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1370/2738] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Aug 30 02:10:49 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 146 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 02:10:49.877981 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07ef4305-6dc3-479b-b3ed-c4e4d36002c1 None None] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:10:49.879713 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-07ef4305-6dc3-479b-b3ed-c4e4d36002c1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:10:49.922120 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:10:49.961076 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07ef4305-6dc3-479b-b3ed-c4e4d36002c1 None None] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:10:49.965892 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c682d9de-551b-4bc3-aeea-ce848d498752 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:10:49.966583 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c682d9de-551b-4bc3-aeea-ce848d498752 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:10:49.966862 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c682d9de-551b-4bc3-aeea-ce848d498752 admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:10:49.967130 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c682d9de-551b-4bc3-aeea-ce848d498752 admin admin] PATH_INFO: `/v3/users/ad16b4f1dfc94571bfcd4554ff588c86` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:10:49.968252 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c682d9de-551b-4bc3-aeea-ce848d498752 admin admin] RBAC: Authorizing `identity:delete_user(user_id=ad16b4f1dfc94571bfcd4554ff588c86)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:10:49.971025 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c682d9de-551b-4bc3-aeea-ce848d498752 admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:10:50.249754 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-c682d9de-551b-4bc3-aeea-ce848d498752 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'ad16b4f1dfc94571bfcd4554ff588c86'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:10:50.250472 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-c682d9de-551b-4bc3-aeea-ce848d498752 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'ad16b4f1dfc94571bfcd4554ff588c86'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:10:50.338228 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-c682d9de-551b-4bc3-aeea-ce848d498752 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'ad16b4f1dfc94571bfcd4554ff588c86'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:10:50.338844 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-c682d9de-551b-4bc3-aeea-ce848d498752 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'ad16b4f1dfc94571bfcd4554ff588c86'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:10:50.346353 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-c682d9de-551b-4bc3-aeea-ce848d498752 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'ad16b4f1dfc94571bfcd4554ff588c86'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:10:50.365329 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1369/2739] 10.222.0.27 () {58 vars in 1224 bytes} [Tue Aug 30 02:10:49 2022] DELETE /identity/v3/users/ad16b4f1dfc94571bfcd4554ff588c86 => generated 0 bytes in 489 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 02:10:50.668046 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cea44e9-d1f6-432a-81c0-377c11c8a4ae service cinder] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:10:50.692034 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cea44e9-d1f6-432a-81c0-377c11c8a4ae service cinder] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:10:50.778657 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60e83887-73ba-4500-a705-eb1da223e0f3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:10:50.779394 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60e83887-73ba-4500-a705-eb1da223e0f3 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:10:50.779761 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60e83887-73ba-4500-a705-eb1da223e0f3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:10:50.780091 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60e83887-73ba-4500-a705-eb1da223e0f3 admin admin] PATH_INFO: `/v3/projects/dbcd3b31535f46308cd1a15fd66538f0` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:10:50.781310 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60e83887-73ba-4500-a705-eb1da223e0f3 admin admin] RBAC: Authorizing `identity:delete_project(project_id=dbcd3b31535f46308cd1a15fd66538f0)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:10:50.783431 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60e83887-73ba-4500-a705-eb1da223e0f3 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:10:50.948336 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-60e83887-73ba-4500-a705-eb1da223e0f3 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'dbcd3b31535f46308cd1a15fd66538f0', 'request_id': 'req-60e83887-73ba-4500-a705-eb1da223e0f3'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:10:50.954714 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-60e83887-73ba-4500-a705-eb1da223e0f3 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'dbcd3b31535f46308cd1a15fd66538f0', 'request_id': 'req-60e83887-73ba-4500-a705-eb1da223e0f3'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:10:50.961961 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-60e83887-73ba-4500-a705-eb1da223e0f3 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'dbcd3b31535f46308cd1a15fd66538f0', 'request_id': 'req-60e83887-73ba-4500-a705-eb1da223e0f3'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:10:51.096042 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-60e83887-73ba-4500-a705-eb1da223e0f3 admin admin] The token cache is being invalidate because project dbcd3b31535f46308cd1a15fd66538f0 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81577) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 30 02:10:51.096323 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-60e83887-73ba-4500-a705-eb1da223e0f3 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:10:51.096955 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-60e83887-73ba-4500-a705-eb1da223e0f3 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:10:51.098637 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1371/2740] 10.222.0.27 () {58 vars in 1233 bytes} [Tue Aug 30 02:10:50 2022] DELETE /identity/v3/projects/dbcd3b31535f46308cd1a15fd66538f0 => generated 0 bytes in 431 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 02:10:51.105708 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c469f09-c71e-4df1-9ba9-0db4932175ad None None] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:10:51.106211 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c469f09-c71e-4df1-9ba9-0db4932175ad None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:10:51.106451 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c469f09-c71e-4df1-9ba9-0db4932175ad None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:10:51.127845 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.auth.core [None req-5c469f09-c71e-4df1-9ba9-0db4932175ad None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81578) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:10:51.129148 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-5c469f09-c71e-4df1-9ba9-0db4932175ad None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:10:51.208936 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1370/2741] 10.222.0.27 () {58 vars in 958 bytes} [Tue Aug 30 02:10:51 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 30 02:10:51.216684 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a2efdb9-4519-45ec-8a5c-8f8137370371 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:10:51.217160 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a2efdb9-4519-45ec-8a5c-8f8137370371 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:10:51.217660 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a2efdb9-4519-45ec-8a5c-8f8137370371 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:10:51.238106 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:10:51.245023 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-8a2efdb9-4519-45ec-8a5c-8f8137370371 None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:10:51.246949 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-8a2efdb9-4519-45ec-8a5c-8f8137370371 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:10:51.264049 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1372/2742] 10.222.0.27 () {58 vars in 958 bytes} [Tue Aug 30 02:10:51 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 48 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 02:10:51.265005 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c469f09-c71e-4df1-9ba9-0db4932175ad None None] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:10:51.281348 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c469f09-c71e-4df1-9ba9-0db4932175ad None None] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:10:51.292898 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0478c665-fc9c-4213-9b3e-6a4c82837bf4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:10:51.293703 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0478c665-fc9c-4213-9b3e-6a4c82837bf4 admin admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:10:51.294528 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0478c665-fc9c-4213-9b3e-6a4c82837bf4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:10:51.294659 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0478c665-fc9c-4213-9b3e-6a4c82837bf4 admin admin] PATH_INFO: `/v3/domains` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:10:51.295529 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0478c665-fc9c-4213-9b3e-6a4c82837bf4 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81578) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 02:10:51.295834 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0478c665-fc9c-4213-9b3e-6a4c82837bf4 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:10:51.297591 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0478c665-fc9c-4213-9b3e-6a4c82837bf4 admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:10:51.301368 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a2efdb9-4519-45ec-8a5c-8f8137370371 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:10:51.303266 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-8a2efdb9-4519-45ec-8a5c-8f8137370371 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:10:51.311414 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1371/2743] 10.222.0.27 () {58 vars in 1153 bytes} [Tue Aug 30 02:10:51 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 47 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 02:10:51.316185 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0478c665-fc9c-4213-9b3e-6a4c82837bf4 admin admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:10:51.326562 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0478c665-fc9c-4213-9b3e-6a4c82837bf4 admin admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:10:51.331964 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43c7e710-d3fc-429f-83d1-928590c4ac0c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:10:51.332572 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43c7e710-d3fc-429f-83d1-928590c4ac0c admin admin] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:10:51.332852 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43c7e710-d3fc-429f-83d1-928590c4ac0c admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:10:51.333129 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43c7e710-d3fc-429f-83d1-928590c4ac0c admin admin] PATH_INFO: `/v3/projects` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:10:51.333803 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43c7e710-d3fc-429f-83d1-928590c4ac0c admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:10:51.336360 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43c7e710-d3fc-429f-83d1-928590c4ac0c admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:10:51.372376 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a2efdb9-4519-45ec-8a5c-8f8137370371 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:10:51.376199 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2963c91f-f6f0-4599-bc9b-1179fd497676 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ab936b3767a8417aa80c8c8f1b84b667', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4562f3951fbe465280073e60a11e44e2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:10:51.376838 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2963c91f-f6f0-4599-bc9b-1179fd497676 service cinder] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:10:51.377447 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2963c91f-f6f0-4599-bc9b-1179fd497676 service cinder] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:10:51.377754 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2963c91f-f6f0-4599-bc9b-1179fd497676 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:10:51.379540 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-2963c91f-f6f0-4599-bc9b-1179fd497676 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:10:51.440044 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1372/2744] 10.222.0.27 () {60 vars in 1153 bytes} [Tue Aug 30 02:10:51 2022] POST /identity/v3/projects => generated 379 bytes in 124 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 30 02:10:51.445176 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43c7e710-d3fc-429f-83d1-928590c4ac0c admin admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:10:51.448106 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2963c91f-f6f0-4599-bc9b-1179fd497676 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:10:51.448106 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2963c91f-f6f0-4599-bc9b-1179fd497676 service cinder] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:10:51.457521 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1373/2745] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Aug 30 02:10:51 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 157 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 30 02:10:51.460202 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=88,keystone:DELETE=10,keystone:INSERT=2 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:10:51.515799 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43c7e710-d3fc-429f-83d1-928590c4ac0c admin admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:10:51.521579 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c225d42-099b-46ec-b213-136d739e877e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:10:51.522076 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c225d42-099b-46ec-b213-136d739e877e admin admin] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:10:51.522352 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c225d42-099b-46ec-b213-136d739e877e admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:10:51.522620 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c225d42-099b-46ec-b213-136d739e877e admin admin] PATH_INFO: `/v3/users` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:10:51.523288 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c225d42-099b-46ec-b213-136d739e877e admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:10:51.525196 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c225d42-099b-46ec-b213-136d739e877e admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:10:51.636391 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1373/2746] 10.222.0.27 () {60 vars in 1144 bytes} [Tue Aug 30 02:10:51 2022] POST /identity/v3/users => generated 290 bytes in 191 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 02:10:51.639663 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2963c91f-f6f0-4599-bc9b-1179fd497676 service cinder] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:10:51.655507 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2963c91f-f6f0-4599-bc9b-1179fd497676 service cinder] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:10:51.661914 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35ed1028-dfca-4316-b3c8-f708acb2652e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:10:51.665187 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35ed1028-dfca-4316-b3c8-f708acb2652e admin admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:10:51.665187 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35ed1028-dfca-4316-b3c8-f708acb2652e admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:10:51.665187 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35ed1028-dfca-4316-b3c8-f708acb2652e admin admin] PATH_INFO: `/v3/roles` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:10:51.665187 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35ed1028-dfca-4316-b3c8-f708acb2652e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:10:51.666132 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35ed1028-dfca-4316-b3c8-f708acb2652e admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:10:51.686011 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1374/2747] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Aug 30 02:10:51 2022] GET /identity/v3/roles => generated 1420 bytes in 47 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 02:10:51.690533 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c225d42-099b-46ec-b213-136d739e877e admin admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:10:51.709393 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c225d42-099b-46ec-b213-136d739e877e admin admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:10:51.714548 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f5cd3d8-fbff-40fd-936c-6b4e0ddf4757 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:10:51.715359 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f5cd3d8-fbff-40fd-936c-6b4e0ddf4757 admin admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:10:51.715890 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f5cd3d8-fbff-40fd-936c-6b4e0ddf4757 admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:10:51.728813 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f5cd3d8-fbff-40fd-936c-6b4e0ddf4757 admin admin] PATH_INFO: `/v3/roles` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:10:51.729772 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f5cd3d8-fbff-40fd-936c-6b4e0ddf4757 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:10:51.732603 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f5cd3d8-fbff-40fd-936c-6b4e0ddf4757 admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:10:51.748606 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1374/2748] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Aug 30 02:10:51 2022] GET /identity/v3/roles => generated 1420 bytes in 56 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 30 02:10:51.750192 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35ed1028-dfca-4316-b3c8-f708acb2652e admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:10:51.761911 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35ed1028-dfca-4316-b3c8-f708acb2652e admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:10:51.767212 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85bf837e-e415-4042-bb73-9ceb761b69d1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:10:51.767944 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85bf837e-e415-4042-bb73-9ceb761b69d1 admin admin] REQUEST_METHOD: `PUT` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:10:51.768306 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85bf837e-e415-4042-bb73-9ceb761b69d1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:10:51.768652 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85bf837e-e415-4042-bb73-9ceb761b69d1 admin admin] PATH_INFO: `/v3/projects/fa2e7ced1148461faedcd0e1856bae95/users/40e0086c1b924e7f86bd814bb45be3d9/roles/1fb81afcb9f245f3b989add94598a063` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:10:51.785681 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85bf837e-e415-4042-bb73-9ceb761b69d1 admin admin] RBAC: Authorizing `identity:create_grant(project_id=fa2e7ced1148461faedcd0e1856bae95, user_id=40e0086c1b924e7f86bd814bb45be3d9, role_id=1fb81afcb9f245f3b989add94598a063)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:10:51.788338 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85bf837e-e415-4042-bb73-9ceb761b69d1 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:10:51.868041 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1375/2749] 10.222.0.27 () {60 vars in 1483 bytes} [Tue Aug 30 02:10:51 2022] PUT /identity/v3/projects/fa2e7ced1148461faedcd0e1856bae95/users/40e0086c1b924e7f86bd814bb45be3d9/roles/1fb81afcb9f245f3b989add94598a063 => generated 0 bytes in 118 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 02:10:51.873837 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d785b2be-b5dd-405f-a2ec-6cb4a4704361 None None] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:10:51.874230 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d785b2be-b5dd-405f-a2ec-6cb4a4704361 None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:10:51.874623 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d785b2be-b5dd-405f-a2ec-6cb4a4704361 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:10:51.900341 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.auth.core [None req-d785b2be-b5dd-405f-a2ec-6cb4a4704361 None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81578) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:10:51.901573 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-d785b2be-b5dd-405f-a2ec-6cb4a4704361 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:10:51.965107 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1375/2750] 10.222.0.27 () {58 vars in 958 bytes} [Tue Aug 30 02:10:51 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 02:10:51.975236 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85bf837e-e415-4042-bb73-9ceb761b69d1 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:10:51.977003 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-85bf837e-e415-4042-bb73-9ceb761b69d1 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:10:52.035502 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85bf837e-e415-4042-bb73-9ceb761b69d1 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:10:52.039246 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6738448e-fe33-4aa3-8cfd-976398538769 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f3ea1580c56f4814a92bbb2acf542905', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4562f3951fbe465280073e60a11e44e2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:10:52.039887 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6738448e-fe33-4aa3-8cfd-976398538769 service neutron] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:10:52.040163 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6738448e-fe33-4aa3-8cfd-976398538769 service neutron] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:10:52.040430 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6738448e-fe33-4aa3-8cfd-976398538769 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:10:52.050328 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6738448e-fe33-4aa3-8cfd-976398538769 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:10:52.051758 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6738448e-fe33-4aa3-8cfd-976398538769 service neutron] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:10:52.061556 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1376/2751] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Aug 30 02:10:51 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 87 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 02:10:53.080322 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d785b2be-b5dd-405f-a2ec-6cb4a4704361 None None] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:10:53.147901 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d785b2be-b5dd-405f-a2ec-6cb4a4704361 None None] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:10:53.151609 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-884a6047-25ca-47f1-a343-afab8ca5e3c5 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ab936b3767a8417aa80c8c8f1b84b667', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4562f3951fbe465280073e60a11e44e2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:10:53.152213 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-884a6047-25ca-47f1-a343-afab8ca5e3c5 service cinder] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:10:53.152486 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-884a6047-25ca-47f1-a343-afab8ca5e3c5 service cinder] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:10:53.152758 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-884a6047-25ca-47f1-a343-afab8ca5e3c5 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:10:53.154501 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-884a6047-25ca-47f1-a343-afab8ca5e3c5 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:10:53.206825 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:10:53.233531 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-884a6047-25ca-47f1-a343-afab8ca5e3c5 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:10:53.235503 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-884a6047-25ca-47f1-a343-afab8ca5e3c5 service cinder] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:10:53.245815 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1376/2752] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Aug 30 02:10:53 2022] GET /identity/v3/auth/tokens => generated 3319 bytes in 167 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 02:10:56.602649 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0b62dae-66b2-4276-99d1-3a7f577c0f57 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:10:56.603154 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0b62dae-66b2-4276-99d1-3a7f577c0f57 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:10:56.603245 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0b62dae-66b2-4276-99d1-3a7f577c0f57 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:10:56.634430 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-e0b62dae-66b2-4276-99d1-3a7f577c0f57 None None] MFA Rules not processed for user `3eb33b1b787447daa68b651f9686574a`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:10:56.635697 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-e0b62dae-66b2-4276-99d1-3a7f577c0f57 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:10:56.657152 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:10:56.700206 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1377/2753] 10.222.0.27 () {58 vars in 958 bytes} [Tue Aug 30 02:10:56 2022] POST /identity/v3/auth/tokens => generated 3266 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 02:10:58.070263 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-884a6047-25ca-47f1-a343-afab8ca5e3c5 service cinder] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:10:58.075140 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-884a6047-25ca-47f1-a343-afab8ca5e3c5 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:10:58.097059 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-884a6047-25ca-47f1-a343-afab8ca5e3c5 service cinder] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:10:58.102367 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-624b406d-2765-4625-a631-ceb144e4ac67 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:10:58.103339 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-624b406d-2765-4625-a631-ceb144e4ac67 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:10:58.103805 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-624b406d-2765-4625-a631-ceb144e4ac67 admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:10:58.104247 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-624b406d-2765-4625-a631-ceb144e4ac67 admin admin] PATH_INFO: `/v3/users/8922516b033d4e24b2ec8d0deb432267` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:10:58.106276 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-624b406d-2765-4625-a631-ceb144e4ac67 admin admin] RBAC: Authorizing `identity:delete_user(user_id=8922516b033d4e24b2ec8d0deb432267)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:10:58.108361 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-624b406d-2765-4625-a631-ceb144e4ac67 admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:10:58.202606 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e211b03-34a7-403c-9c2c-8674045ab60e None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:10:58.203084 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e211b03-34a7-403c-9c2c-8674045ab60e None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:10:58.203317 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e211b03-34a7-403c-9c2c-8674045ab60e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:10:58.241829 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-8e211b03-34a7-403c-9c2c-8674045ab60e None None] MFA Rules not processed for user `40e0086c1b924e7f86bd814bb45be3d9`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:10:58.243389 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-8e211b03-34a7-403c-9c2c-8674045ab60e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:10:58.326417 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1378/2754] 10.222.0.27 () {58 vars in 958 bytes} [Tue Aug 30 02:10:58 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 125 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 30 02:10:58.330977 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e211b03-34a7-403c-9c2c-8674045ab60e None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:10:58.342777 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e211b03-34a7-403c-9c2c-8674045ab60e None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:10:58.348072 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c87dc01-2a1a-4733-bc0d-9a9d04f2909f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:10:58.348700 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c87dc01-2a1a-4733-bc0d-9a9d04f2909f admin admin] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:10:58.348979 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c87dc01-2a1a-4733-bc0d-9a9d04f2909f admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:10:58.349254 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c87dc01-2a1a-4733-bc0d-9a9d04f2909f admin admin] PATH_INFO: `/v3/projects` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:10:58.350584 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c87dc01-2a1a-4733-bc0d-9a9d04f2909f admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:10:58.352495 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c87dc01-2a1a-4733-bc0d-9a9d04f2909f admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:10:58.368959 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-624b406d-2765-4625-a631-ceb144e4ac67 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '8922516b033d4e24b2ec8d0deb432267'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:10:58.370184 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-624b406d-2765-4625-a631-ceb144e4ac67 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '8922516b033d4e24b2ec8d0deb432267'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:10:58.446825 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1379/2755] 10.222.0.27 () {60 vars in 1153 bytes} [Tue Aug 30 02:10:58 2022] POST /identity/v3/projects => generated 377 bytes in 116 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 30 02:10:58.450685 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c87dc01-2a1a-4733-bc0d-9a9d04f2909f admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:10:58.470999 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-624b406d-2765-4625-a631-ceb144e4ac67 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '8922516b033d4e24b2ec8d0deb432267'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:10:58.472031 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-624b406d-2765-4625-a631-ceb144e4ac67 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '8922516b033d4e24b2ec8d0deb432267'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:10:58.479954 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-624b406d-2765-4625-a631-ceb144e4ac67 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '8922516b033d4e24b2ec8d0deb432267'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:10:58.498570 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1377/2756] 10.222.0.27 () {58 vars in 1224 bytes} [Tue Aug 30 02:10:58 2022] DELETE /identity/v3/users/8922516b033d4e24b2ec8d0deb432267 => generated 0 bytes in 429 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 02:10:58.503801 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cacc5ace-d5cc-45aa-8b35-d1e7a643e8e3 None None] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:10:58.504084 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cacc5ace-d5cc-45aa-8b35-d1e7a643e8e3 None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:10:58.504389 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cacc5ace-d5cc-45aa-8b35-d1e7a643e8e3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:10:58.517474 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c87dc01-2a1a-4733-bc0d-9a9d04f2909f admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:10:58.524459 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.auth.core [None req-cacc5ace-d5cc-45aa-8b35-d1e7a643e8e3 None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81578) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:10:58.525689 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-cacc5ace-d5cc-45aa-8b35-d1e7a643e8e3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:10:58.558596 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:10:58.575234 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b09f455a-095f-4893-81de-c632f22c62f0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:10:58.575846 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b09f455a-095f-4893-81de-c632f22c62f0 admin admin] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:10:58.576119 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b09f455a-095f-4893-81de-c632f22c62f0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:10:58.576385 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b09f455a-095f-4893-81de-c632f22c62f0 admin admin] PATH_INFO: `/v3/users` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:10:58.577011 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b09f455a-095f-4893-81de-c632f22c62f0 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:10:58.578999 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b09f455a-095f-4893-81de-c632f22c62f0 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:10:58.585041 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1378/2757] 10.222.0.27 () {58 vars in 958 bytes} [Tue Aug 30 02:10:58 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 82 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 02:10:58.595453 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cacc5ace-d5cc-45aa-8b35-d1e7a643e8e3 None None] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:10:58.596810 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-cacc5ace-d5cc-45aa-8b35-d1e7a643e8e3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:10:58.660746 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cacc5ace-d5cc-45aa-8b35-d1e7a643e8e3 None None] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:10:58.663935 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1380/2758] 10.222.0.27 () {60 vars in 1144 bytes} [Tue Aug 30 02:10:58 2022] POST /identity/v3/users => generated 295 bytes in 214 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 30 02:10:58.664909 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b09f455a-095f-4893-81de-c632f22c62f0 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:10:58.665181 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3b399fd-7026-4a54-9f14-ef0975a360d7 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f3ea1580c56f4814a92bbb2acf542905', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4562f3951fbe465280073e60a11e44e2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:10:58.665943 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3b399fd-7026-4a54-9f14-ef0975a360d7 service neutron] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:10:58.666217 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3b399fd-7026-4a54-9f14-ef0975a360d7 service neutron] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:10:58.666487 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3b399fd-7026-4a54-9f14-ef0975a360d7 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:10:58.666932 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-b09f455a-095f-4893-81de-c632f22c62f0 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:10:58.676981 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3b399fd-7026-4a54-9f14-ef0975a360d7 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:10:58.678423 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3b399fd-7026-4a54-9f14-ef0975a360d7 service neutron] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:10:58.678757 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b09f455a-095f-4893-81de-c632f22c62f0 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:10:58.682616 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:10:58.683401 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1458aea9-aa9e-4703-98ea-f90348c44276 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:10:58.683990 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1458aea9-aa9e-4703-98ea-f90348c44276 admin admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:10:58.684264 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1458aea9-aa9e-4703-98ea-f90348c44276 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:10:58.684533 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1458aea9-aa9e-4703-98ea-f90348c44276 admin admin] PATH_INFO: `/v3/roles` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:10:58.685164 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1458aea9-aa9e-4703-98ea-f90348c44276 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:10:58.686856 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1458aea9-aa9e-4703-98ea-f90348c44276 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:10:58.688941 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1379/2759] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Aug 30 02:10:58 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 94 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 30 02:10:58.697837 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1381/2760] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Aug 30 02:10:58 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 30 02:10:58.702060 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3b399fd-7026-4a54-9f14-ef0975a360d7 service neutron] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:10:58.712780 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3b399fd-7026-4a54-9f14-ef0975a360d7 service neutron] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:10:58.717718 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fca0e64b-0b8a-4833-a13d-1b4679fa62fc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:10:58.718443 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fca0e64b-0b8a-4833-a13d-1b4679fa62fc admin admin] REQUEST_METHOD: `PUT` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:10:58.718651 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fca0e64b-0b8a-4833-a13d-1b4679fa62fc admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:10:58.718925 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fca0e64b-0b8a-4833-a13d-1b4679fa62fc admin admin] PATH_INFO: `/v3/projects/821c7a0aa57a472398a82afbc16a598e/users/4c2c70735cff4748866f0dab613d24a0/roles/ead5fac86a0747ea8823a82efcea8ed7` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:10:58.735985 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fca0e64b-0b8a-4833-a13d-1b4679fa62fc admin admin] RBAC: Authorizing `identity:create_grant(project_id=821c7a0aa57a472398a82afbc16a598e, user_id=4c2c70735cff4748866f0dab613d24a0, role_id=ead5fac86a0747ea8823a82efcea8ed7)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:10:58.738860 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fca0e64b-0b8a-4833-a13d-1b4679fa62fc admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:10:58.870938 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1380/2761] 10.222.0.27 () {60 vars in 1483 bytes} [Tue Aug 30 02:10:58 2022] PUT /identity/v3/projects/821c7a0aa57a472398a82afbc16a598e/users/4c2c70735cff4748866f0dab613d24a0/roles/ead5fac86a0747ea8823a82efcea8ed7 => generated 0 bytes in 169 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 30 02:10:58.875742 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1458aea9-aa9e-4703-98ea-f90348c44276 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:10:58.889613 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1458aea9-aa9e-4703-98ea-f90348c44276 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:10:58.987032 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-084d20a2-ecbc-430c-976c-38ce5a8de739 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:10:58.987783 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-084d20a2-ecbc-430c-976c-38ce5a8de739 admin admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:10:58.988718 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-084d20a2-ecbc-430c-976c-38ce5a8de739 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:10:58.989063 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-084d20a2-ecbc-430c-976c-38ce5a8de739 admin admin] PATH_INFO: `/v3/roles` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:10:58.989808 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-084d20a2-ecbc-430c-976c-38ce5a8de739 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:10:58.992316 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-084d20a2-ecbc-430c-976c-38ce5a8de739 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:10:59.009842 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1382/2762] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Aug 30 02:10:58 2022] GET /identity/v3/roles => generated 1420 bytes in 135 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 02:10:59.013849 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fca0e64b-0b8a-4833-a13d-1b4679fa62fc admin admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:10:59.024860 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fca0e64b-0b8a-4833-a13d-1b4679fa62fc admin admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:10:59.034149 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77caa6a7-1b13-4b28-971e-cd85acbdd771 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:10:59.036267 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77caa6a7-1b13-4b28-971e-cd85acbdd771 admin admin] REQUEST_METHOD: `PUT` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:10:59.037111 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77caa6a7-1b13-4b28-971e-cd85acbdd771 admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:10:59.038391 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77caa6a7-1b13-4b28-971e-cd85acbdd771 admin admin] PATH_INFO: `/v3/projects/821c7a0aa57a472398a82afbc16a598e/users/4c2c70735cff4748866f0dab613d24a0/roles/ead5fac86a0747ea8823a82efcea8ed7` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:10:59.043142 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77caa6a7-1b13-4b28-971e-cd85acbdd771 admin admin] RBAC: Authorizing `identity:create_grant(project_id=821c7a0aa57a472398a82afbc16a598e, user_id=4c2c70735cff4748866f0dab613d24a0, role_id=ead5fac86a0747ea8823a82efcea8ed7)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:10:59.046893 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77caa6a7-1b13-4b28-971e-cd85acbdd771 admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:10:59.062982 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1381/2763] 10.222.0.27 () {60 vars in 1483 bytes} [Tue Aug 30 02:10:59 2022] PUT /identity/v3/projects/821c7a0aa57a472398a82afbc16a598e/users/4c2c70735cff4748866f0dab613d24a0/roles/ead5fac86a0747ea8823a82efcea8ed7 => generated 0 bytes in 50 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 02:10:59.152324 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-084d20a2-ecbc-430c-976c-38ce5a8de739 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:10:59.154741 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-084d20a2-ecbc-430c-976c-38ce5a8de739 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:10:59.189966 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-084d20a2-ecbc-430c-976c-38ce5a8de739 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:10:59.264112 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83d7fea8-9c5b-46a5-8ea3-9c531b57f1d4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:10:59.265828 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83d7fea8-9c5b-46a5-8ea3-9c531b57f1d4 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:10:59.266463 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83d7fea8-9c5b-46a5-8ea3-9c531b57f1d4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:10:59.269386 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83d7fea8-9c5b-46a5-8ea3-9c531b57f1d4 admin admin] PATH_INFO: `/v3/projects/c25441166f7246b4b999cdd7383ad358` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:10:59.269386 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83d7fea8-9c5b-46a5-8ea3-9c531b57f1d4 admin admin] RBAC: Authorizing `identity:delete_project(project_id=c25441166f7246b4b999cdd7383ad358)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:10:59.271255 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83d7fea8-9c5b-46a5-8ea3-9c531b57f1d4 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:10:59.408273 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-83d7fea8-9c5b-46a5-8ea3-9c531b57f1d4 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'c25441166f7246b4b999cdd7383ad358', 'request_id': 'req-83d7fea8-9c5b-46a5-8ea3-9c531b57f1d4'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:10:59.414518 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-83d7fea8-9c5b-46a5-8ea3-9c531b57f1d4 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'c25441166f7246b4b999cdd7383ad358', 'request_id': 'req-83d7fea8-9c5b-46a5-8ea3-9c531b57f1d4'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:10:59.415802 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=91,keystone:INSERT=3,keystone:DELETE=6 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:10:59.422355 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-83d7fea8-9c5b-46a5-8ea3-9c531b57f1d4 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'c25441166f7246b4b999cdd7383ad358', 'request_id': 'req-83d7fea8-9c5b-46a5-8ea3-9c531b57f1d4'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:10:59.527126 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-83d7fea8-9c5b-46a5-8ea3-9c531b57f1d4 admin admin] The token cache is being invalidate because project c25441166f7246b4b999cdd7383ad358 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81577) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 30 02:10:59.527479 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-83d7fea8-9c5b-46a5-8ea3-9c531b57f1d4 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:10:59.528733 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-83d7fea8-9c5b-46a5-8ea3-9c531b57f1d4 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:10:59.530290 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1383/2764] 10.222.0.27 () {58 vars in 1233 bytes} [Tue Aug 30 02:10:59 2022] DELETE /identity/v3/projects/c25441166f7246b4b999cdd7383ad358 => generated 0 bytes in 379 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 02:10:59.535632 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77caa6a7-1b13-4b28-971e-cd85acbdd771 admin admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:10:59.537367 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-77caa6a7-1b13-4b28-971e-cd85acbdd771 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:10:59.551534 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77caa6a7-1b13-4b28-971e-cd85acbdd771 admin admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:10:59.615882 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a09e1c97-ffe2-4d43-9719-be0affcc38ad admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:10:59.616127 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a09e1c97-ffe2-4d43-9719-be0affcc38ad admin admin] REQUEST_METHOD: `DELETE` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:10:59.616486 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a09e1c97-ffe2-4d43-9719-be0affcc38ad admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:10:59.616765 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a09e1c97-ffe2-4d43-9719-be0affcc38ad admin admin] PATH_INFO: `/v3/users/7e93952263124221b28466603da9aa65` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:10:59.617958 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a09e1c97-ffe2-4d43-9719-be0affcc38ad admin admin] RBAC: Authorizing `identity:delete_user(user_id=7e93952263124221b28466603da9aa65)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:10:59.619888 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a09e1c97-ffe2-4d43-9719-be0affcc38ad admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:10:59.861960 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-a09e1c97-ffe2-4d43-9719-be0affcc38ad admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '7e93952263124221b28466603da9aa65'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:10:59.870739 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-a09e1c97-ffe2-4d43-9719-be0affcc38ad admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '7e93952263124221b28466603da9aa65'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:10:59.973536 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-a09e1c97-ffe2-4d43-9719-be0affcc38ad admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '7e93952263124221b28466603da9aa65'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:10:59.974824 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-a09e1c97-ffe2-4d43-9719-be0affcc38ad admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '7e93952263124221b28466603da9aa65'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:10:59.987169 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-a09e1c97-ffe2-4d43-9719-be0affcc38ad admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '7e93952263124221b28466603da9aa65'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:11:00.010238 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1382/2765] 10.222.0.27 () {58 vars in 1224 bytes} [Tue Aug 30 02:10:59 2022] DELETE /identity/v3/users/7e93952263124221b28466603da9aa65 => generated 0 bytes in 475 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 02:11:00.784543 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83d7fea8-9c5b-46a5-8ea3-9c531b57f1d4 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:11:00.803279 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83d7fea8-9c5b-46a5-8ea3-9c531b57f1d4 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:11:00.903793 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62027a5a-bba9-43f6-9b47-c93a978a33e7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:11:00.904705 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62027a5a-bba9-43f6-9b47-c93a978a33e7 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:11:00.904993 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62027a5a-bba9-43f6-9b47-c93a978a33e7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:11:00.905271 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62027a5a-bba9-43f6-9b47-c93a978a33e7 admin admin] PATH_INFO: `/v3/projects/ee3fa3bd0a1d471d9fddc60930e62bb3` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:11:00.906587 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62027a5a-bba9-43f6-9b47-c93a978a33e7 admin admin] RBAC: Authorizing `identity:delete_project(project_id=ee3fa3bd0a1d471d9fddc60930e62bb3)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:11:00.909175 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62027a5a-bba9-43f6-9b47-c93a978a33e7 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:11:01.021343 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-62027a5a-bba9-43f6-9b47-c93a978a33e7 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'ee3fa3bd0a1d471d9fddc60930e62bb3', 'request_id': 'req-62027a5a-bba9-43f6-9b47-c93a978a33e7'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:11:01.029824 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-62027a5a-bba9-43f6-9b47-c93a978a33e7 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'ee3fa3bd0a1d471d9fddc60930e62bb3', 'request_id': 'req-62027a5a-bba9-43f6-9b47-c93a978a33e7'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:11:01.044448 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-62027a5a-bba9-43f6-9b47-c93a978a33e7 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'ee3fa3bd0a1d471d9fddc60930e62bb3', 'request_id': 'req-62027a5a-bba9-43f6-9b47-c93a978a33e7'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:11:01.144653 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-62027a5a-bba9-43f6-9b47-c93a978a33e7 admin admin] The token cache is being invalidate because project ee3fa3bd0a1d471d9fddc60930e62bb3 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81577) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 30 02:11:01.145705 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-62027a5a-bba9-43f6-9b47-c93a978a33e7 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:11:01.146215 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-62027a5a-bba9-43f6-9b47-c93a978a33e7 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:11:01.153388 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1384/2766] 10.222.0.27 () {58 vars in 1233 bytes} [Tue Aug 30 02:11:00 2022] DELETE /identity/v3/projects/ee3fa3bd0a1d471d9fddc60930e62bb3 => generated 0 bytes in 366 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 02:11:01.154101 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a09e1c97-ffe2-4d43-9719-be0affcc38ad admin admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:11:01.155799 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-a09e1c97-ffe2-4d43-9719-be0affcc38ad admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:11:01.193241 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a09e1c97-ffe2-4d43-9719-be0affcc38ad admin admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:11:01.271115 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=87,keystone:INSERT=3,keystone:DELETE=10 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:11:01.304887 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47480a87-3f5c-4120-9a39-b1b99d2a8a71 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:11:01.305966 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47480a87-3f5c-4120-9a39-b1b99d2a8a71 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:11:01.306456 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47480a87-3f5c-4120-9a39-b1b99d2a8a71 admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:11:01.306911 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47480a87-3f5c-4120-9a39-b1b99d2a8a71 admin admin] PATH_INFO: `/v3/users/0ed9ee2708c6439a9bd2401310a54cbc` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:11:01.309084 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47480a87-3f5c-4120-9a39-b1b99d2a8a71 admin admin] RBAC: Authorizing `identity:delete_user(user_id=0ed9ee2708c6439a9bd2401310a54cbc)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:11:01.311424 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47480a87-3f5c-4120-9a39-b1b99d2a8a71 admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:11:01.796329 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-47480a87-3f5c-4120-9a39-b1b99d2a8a71 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '0ed9ee2708c6439a9bd2401310a54cbc'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:11:01.800407 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-47480a87-3f5c-4120-9a39-b1b99d2a8a71 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '0ed9ee2708c6439a9bd2401310a54cbc'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:11:01.934790 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-47480a87-3f5c-4120-9a39-b1b99d2a8a71 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '0ed9ee2708c6439a9bd2401310a54cbc'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:11:01.938061 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-47480a87-3f5c-4120-9a39-b1b99d2a8a71 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '0ed9ee2708c6439a9bd2401310a54cbc'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:11:01.976961 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-47480a87-3f5c-4120-9a39-b1b99d2a8a71 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '0ed9ee2708c6439a9bd2401310a54cbc'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:11:02.032323 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1383/2767] 10.222.0.27 () {58 vars in 1224 bytes} [Tue Aug 30 02:11:01 2022] DELETE /identity/v3/users/0ed9ee2708c6439a9bd2401310a54cbc => generated 0 bytes in 878 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 02:11:02.980819 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62027a5a-bba9-43f6-9b47-c93a978a33e7 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:11:02.982372 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-62027a5a-bba9-43f6-9b47-c93a978a33e7 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:11:03.003456 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62027a5a-bba9-43f6-9b47-c93a978a33e7 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:11:03.148231 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1ed2438-d44c-4c83-acdd-a650970c2c10 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:11:03.148231 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1ed2438-d44c-4c83-acdd-a650970c2c10 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:11:03.148231 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1ed2438-d44c-4c83-acdd-a650970c2c10 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:11:03.148231 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1ed2438-d44c-4c83-acdd-a650970c2c10 admin admin] PATH_INFO: `/v3/projects/733a23ea0e3945d9ab688b994ebe1192` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:11:03.148637 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1ed2438-d44c-4c83-acdd-a650970c2c10 admin admin] RBAC: Authorizing `identity:delete_project(project_id=733a23ea0e3945d9ab688b994ebe1192)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:11:03.148903 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1ed2438-d44c-4c83-acdd-a650970c2c10 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:11:03.176319 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=90,keystone:INSERT=2,keystone:DELETE=8 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:11:03.290585 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-e1ed2438-d44c-4c83-acdd-a650970c2c10 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '733a23ea0e3945d9ab688b994ebe1192', 'request_id': 'req-e1ed2438-d44c-4c83-acdd-a650970c2c10'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:11:03.300682 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-e1ed2438-d44c-4c83-acdd-a650970c2c10 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '733a23ea0e3945d9ab688b994ebe1192', 'request_id': 'req-e1ed2438-d44c-4c83-acdd-a650970c2c10'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:11:03.316316 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-e1ed2438-d44c-4c83-acdd-a650970c2c10 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '733a23ea0e3945d9ab688b994ebe1192', 'request_id': 'req-e1ed2438-d44c-4c83-acdd-a650970c2c10'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:11:03.394963 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-e1ed2438-d44c-4c83-acdd-a650970c2c10 admin admin] The token cache is being invalidate because project 733a23ea0e3945d9ab688b994ebe1192 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81577) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 30 02:11:03.395256 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-e1ed2438-d44c-4c83-acdd-a650970c2c10 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:11:03.395946 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-e1ed2438-d44c-4c83-acdd-a650970c2c10 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:11:03.397735 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1385/2768] 10.222.0.27 () {58 vars in 1233 bytes} [Tue Aug 30 02:11:02 2022] DELETE /identity/v3/projects/733a23ea0e3945d9ab688b994ebe1192 => generated 0 bytes in 418 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 02:11:05.814258 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54b9af55-69cd-4bd8-b253-858b8365eb3c None None] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:11:05.815189 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54b9af55-69cd-4bd8-b253-858b8365eb3c None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:11:05.815670 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54b9af55-69cd-4bd8-b253-858b8365eb3c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:11:05.850519 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.auth.core [None req-54b9af55-69cd-4bd8-b253-858b8365eb3c None None] MFA Rules not processed for user `4c2c70735cff4748866f0dab613d24a0`. Rule list: `[]` (Enabled: `True`). {{(pid=81578) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:11:05.851773 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-54b9af55-69cd-4bd8-b253-858b8365eb3c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:11:05.951888 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1384/2769] 10.222.0.27 () {58 vars in 958 bytes} [Tue Aug 30 02:11:05 2022] POST /identity/v3/auth/tokens => generated 3321 bytes in 139 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 02:11:05.968196 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1ed2438-d44c-4c83-acdd-a650970c2c10 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:11:05.970030 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-e1ed2438-d44c-4c83-acdd-a650970c2c10 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:11:06.089071 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1ed2438-d44c-4c83-acdd-a650970c2c10 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:11:06.095470 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e32e1a71-80d7-48a8-adf2-e60cd55ac651 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd64f2e71361644589bec8a06e6eceed1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4562f3951fbe465280073e60a11e44e2', 'project_domain_id': 'default', 'roles': ['member', 'service', 'admin', 'ResellerAdmin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:11:06.096337 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e32e1a71-80d7-48a8-adf2-e60cd55ac651 service nova] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:11:06.096489 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e32e1a71-80d7-48a8-adf2-e60cd55ac651 service nova] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:11:06.096743 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e32e1a71-80d7-48a8-adf2-e60cd55ac651 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:11:06.098999 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-e32e1a71-80d7-48a8-adf2-e60cd55ac651 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:11:06.174759 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e32e1a71-80d7-48a8-adf2-e60cd55ac651 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:11:06.180490 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e32e1a71-80d7-48a8-adf2-e60cd55ac651 service nova] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:11:06.187312 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1386/2770] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Aug 30 02:11:05 2022] GET /identity/v3/auth/tokens => generated 3256 bytes in 220 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 02:11:15.948544 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=69,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:11:16.182377 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=81,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:11:24.607300 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d21455bc-b468-4f51-810d-fc27b87a5019 None None] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:11:24.607852 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d21455bc-b468-4f51-810d-fc27b87a5019 None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:11:24.607852 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d21455bc-b468-4f51-810d-fc27b87a5019 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:11:24.628048 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.auth.core [None req-d21455bc-b468-4f51-810d-fc27b87a5019 None None] MFA Rules not processed for user `40e0086c1b924e7f86bd814bb45be3d9`. Rule list: `[]` (Enabled: `True`). {{(pid=81578) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:11:24.629282 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-d21455bc-b468-4f51-810d-fc27b87a5019 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:11:24.636633 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1385/2771] 10.222.0.27 () {58 vars in 958 bytes} [Tue Aug 30 02:11:24 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 02:11:34.624461 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=3 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:11:45.937256 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e32e1a71-80d7-48a8-adf2-e60cd55ac651 service nova] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:11:45.938677 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-e32e1a71-80d7-48a8-adf2-e60cd55ac651 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:11:45.994310 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e32e1a71-80d7-48a8-adf2-e60cd55ac651 service nova] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:11:45.998028 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e05c019-bfa5-4bd6-bedf-57946d9a8f63 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f3ea1580c56f4814a92bbb2acf542905', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4562f3951fbe465280073e60a11e44e2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:11:45.998630 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e05c019-bfa5-4bd6-bedf-57946d9a8f63 service neutron] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:11:45.998908 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e05c019-bfa5-4bd6-bedf-57946d9a8f63 service neutron] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:11:45.999173 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e05c019-bfa5-4bd6-bedf-57946d9a8f63 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:11:46.008092 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e05c019-bfa5-4bd6-bedf-57946d9a8f63 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:11:46.010079 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e05c019-bfa5-4bd6-bedf-57946d9a8f63 service neutron] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:11:46.019306 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1387/2772] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Aug 30 02:11:45 2022] GET /identity/v3/auth/tokens => generated 3321 bytes in 83 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 02:11:56.015252 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=26 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:11:58.913933 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d21455bc-b468-4f51-810d-fc27b87a5019 None None] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:11:58.915382 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-d21455bc-b468-4f51-810d-fc27b87a5019 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:11:58.928030 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d21455bc-b468-4f51-810d-fc27b87a5019 None None] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:11:58.931545 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42c8c61e-e9e6-4622-904b-bec7773e4aca service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f3ea1580c56f4814a92bbb2acf542905', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4562f3951fbe465280073e60a11e44e2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:11:58.932145 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42c8c61e-e9e6-4622-904b-bec7773e4aca service neutron] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:11:58.932470 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42c8c61e-e9e6-4622-904b-bec7773e4aca service neutron] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:11:58.932828 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42c8c61e-e9e6-4622-904b-bec7773e4aca service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:11:58.934701 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-42c8c61e-e9e6-4622-904b-bec7773e4aca service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:11:58.947857 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42c8c61e-e9e6-4622-904b-bec7773e4aca service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:11:58.949363 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42c8c61e-e9e6-4622-904b-bec7773e4aca service neutron] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:11:58.960482 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1386/2773] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Aug 30 02:11:58 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 47 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 02:12:00.141909 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e05c019-bfa5-4bd6-bedf-57946d9a8f63 service neutron] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:12:00.150734 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e05c019-bfa5-4bd6-bedf-57946d9a8f63 service neutron] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:12:00.160073 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-160f0770-37f7-4209-ae0a-2a8c5b851331 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f3ea1580c56f4814a92bbb2acf542905', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4562f3951fbe465280073e60a11e44e2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:12:00.160073 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-160f0770-37f7-4209-ae0a-2a8c5b851331 service neutron] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:12:00.160073 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-160f0770-37f7-4209-ae0a-2a8c5b851331 service neutron] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:12:00.160073 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-160f0770-37f7-4209-ae0a-2a8c5b851331 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:12:00.160473 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-160f0770-37f7-4209-ae0a-2a8c5b851331 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:12:00.171113 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-160f0770-37f7-4209-ae0a-2a8c5b851331 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:12:00.172504 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-160f0770-37f7-4209-ae0a-2a8c5b851331 service neutron] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:12:00.183864 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1388/2774] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Aug 30 02:12:00 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 42 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 02:12:08.703883 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61ab2eb4-582b-4e54-94ca-be3104c0bd2c None None] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:12:08.704407 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61ab2eb4-582b-4e54-94ca-be3104c0bd2c None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:12:08.704407 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61ab2eb4-582b-4e54-94ca-be3104c0bd2c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:12:08.724138 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.auth.core [None req-61ab2eb4-582b-4e54-94ca-be3104c0bd2c None None] MFA Rules not processed for user `121c374d49a64335ab74a431cfbc1320`. Rule list: `[]` (Enabled: `True`). {{(pid=81578) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:12:08.725340 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-61ab2eb4-582b-4e54-94ca-be3104c0bd2c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:12:08.783274 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1387/2775] 10.222.0.27 () {58 vars in 1009 bytes} [Tue Aug 30 02:12:08 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 80 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 02:12:08.914283 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-160f0770-37f7-4209-ae0a-2a8c5b851331 service neutron] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:12:08.915679 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-160f0770-37f7-4209-ae0a-2a8c5b851331 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:12:08.982655 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-160f0770-37f7-4209-ae0a-2a8c5b851331 service neutron] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:12:08.987670 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba36f603-7463-4e31-ae0e-b3b02a5aad80 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b4f0008fde424f22bc1f7307cf422774', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4562f3951fbe465280073e60a11e44e2', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:12:08.988262 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba36f603-7463-4e31-ae0e-b3b02a5aad80 service swift] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:12:08.988540 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba36f603-7463-4e31-ae0e-b3b02a5aad80 service swift] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:12:08.988810 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba36f603-7463-4e31-ae0e-b3b02a5aad80 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:12:08.996637 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba36f603-7463-4e31-ae0e-b3b02a5aad80 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:12:08.998261 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba36f603-7463-4e31-ae0e-b3b02a5aad80 service swift] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:12:09.006007 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1389/2776] 10.222.0.27 () {60 vars in 1384 bytes} [Tue Aug 30 02:12:08 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 92 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Aug 30 02:12:18.775960 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=31 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:12:19.002885 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=38 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:12:25.089621 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49b22e97-fd2c-4e8f-b61c-3eed5c0f5997 None None] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:12:25.090192 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49b22e97-fd2c-4e8f-b61c-3eed5c0f5997 None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:12:25.090192 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49b22e97-fd2c-4e8f-b61c-3eed5c0f5997 None None] PATH_INFO: `/` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:12:25.092265 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1388/2777] 10.222.0.27 () {54 vars in 921 bytes} [Tue Aug 30 02:12:25 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 02:12:25.100593 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b89f8570-651b-4423-82ca-dc5a615f93f5 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:12:25.101111 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b89f8570-651b-4423-82ca-dc5a615f93f5 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:12:25.101111 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b89f8570-651b-4423-82ca-dc5a615f93f5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:12:25.140048 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-b89f8570-651b-4423-82ca-dc5a615f93f5 None None] MFA Rules not processed for user `d64f2e71361644589bec8a06e6eceed1`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:12:25.141272 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-b89f8570-651b-4423-82ca-dc5a615f93f5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:12:25.150845 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1390/2778] 10.222.0.27 () {58 vars in 1018 bytes} [Tue Aug 30 02:12:25 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 51 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 02:12:25.164243 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49b22e97-fd2c-4e8f-b61c-3eed5c0f5997 None None] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:12:25.165993 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-49b22e97-fd2c-4e8f-b61c-3eed5c0f5997 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:12:25.223670 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49b22e97-fd2c-4e8f-b61c-3eed5c0f5997 None None] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:12:25.227234 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09ceb37f-c67f-436f-bc5c-448595bfca97 service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3caff62093d748f48596574836e12e42', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4562f3951fbe465280073e60a11e44e2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:12:25.227831 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09ceb37f-c67f-436f-bc5c-448595bfca97 service glance] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:12:25.228105 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09ceb37f-c67f-436f-bc5c-448595bfca97 service glance] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:12:25.228370 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09ceb37f-c67f-436f-bc5c-448595bfca97 service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:12:25.239139 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09ceb37f-c67f-436f-bc5c-448595bfca97 service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:12:25.240544 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09ceb37f-c67f-436f-bc5c-448595bfca97 service glance] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:12:25.250913 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1389/2779] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Aug 30 02:12:25 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 87 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 02:12:25.541897 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8916f89d-807f-4575-94ae-52b00801f5e0 None None] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:12:25.542111 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8916f89d-807f-4575-94ae-52b00801f5e0 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:12:25.542365 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8916f89d-807f-4575-94ae-52b00801f5e0 None None] PATH_INFO: `/` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:12:25.543980 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1391/2780] 10.222.0.27 () {54 vars in 921 bytes} [Tue Aug 30 02:12:25 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 02:12:25.548692 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b24a1e65-f76f-45f1-b3ef-0e2170755e1c None None] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:12:25.549120 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b24a1e65-f76f-45f1-b3ef-0e2170755e1c None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:12:25.549409 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b24a1e65-f76f-45f1-b3ef-0e2170755e1c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:12:25.586831 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.auth.core [None req-b24a1e65-f76f-45f1-b3ef-0e2170755e1c None None] MFA Rules not processed for user `f3ea1580c56f4814a92bbb2acf542905`. Rule list: `[]` (Enabled: `True`). {{(pid=81578) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:12:25.588042 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-b24a1e65-f76f-45f1-b3ef-0e2170755e1c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:12:25.595385 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1390/2781] 10.222.0.27 () {58 vars in 1018 bytes} [Tue Aug 30 02:12:25 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 48 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 02:12:25.607575 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8916f89d-807f-4575-94ae-52b00801f5e0 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:12:25.618302 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8916f89d-807f-4575-94ae-52b00801f5e0 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:12:25.621826 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41de0808-be8d-4661-817a-70bc0f8e151c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f3ea1580c56f4814a92bbb2acf542905', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4562f3951fbe465280073e60a11e44e2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:12:25.622486 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41de0808-be8d-4661-817a-70bc0f8e151c service neutron] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:12:25.622757 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41de0808-be8d-4661-817a-70bc0f8e151c service neutron] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:12:25.623022 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41de0808-be8d-4661-817a-70bc0f8e151c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:12:25.629832 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41de0808-be8d-4661-817a-70bc0f8e151c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:12:25.631231 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41de0808-be8d-4661-817a-70bc0f8e151c service neutron] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:12:25.639534 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1392/2782] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Aug 30 02:12:25 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 33 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 30 02:12:25.724180 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-989d4960-6400-4c00-9dce-096669b93f1c None None] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:12:25.724477 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-989d4960-6400-4c00-9dce-096669b93f1c None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:12:25.724753 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-989d4960-6400-4c00-9dce-096669b93f1c None None] PATH_INFO: `/` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:12:25.726145 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1391/2783] 10.222.0.27 () {54 vars in 921 bytes} [Tue Aug 30 02:12:25 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 30 02:12:25.731137 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b11a9568-5494-48ac-adc7-304339b68558 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:12:25.731449 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b11a9568-5494-48ac-adc7-304339b68558 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:12:25.731762 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b11a9568-5494-48ac-adc7-304339b68558 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:12:25.751600 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-b11a9568-5494-48ac-adc7-304339b68558 None None] MFA Rules not processed for user `d64f2e71361644589bec8a06e6eceed1`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:12:25.752771 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-b11a9568-5494-48ac-adc7-304339b68558 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:12:25.761767 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1393/2784] 10.222.0.27 () {58 vars in 1018 bytes} [Tue Aug 30 02:12:25 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 02:12:25.772577 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-989d4960-6400-4c00-9dce-096669b93f1c None None] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:12:25.784173 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-989d4960-6400-4c00-9dce-096669b93f1c None None] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:12:25.787807 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0571ee65-c41b-4ff9-a142-6ec8359fbd4c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f3ea1580c56f4814a92bbb2acf542905', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4562f3951fbe465280073e60a11e44e2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:12:25.788392 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0571ee65-c41b-4ff9-a142-6ec8359fbd4c service neutron] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:12:25.788667 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0571ee65-c41b-4ff9-a142-6ec8359fbd4c service neutron] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:12:25.788960 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0571ee65-c41b-4ff9-a142-6ec8359fbd4c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:12:25.799066 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0571ee65-c41b-4ff9-a142-6ec8359fbd4c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:12:25.800468 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0571ee65-c41b-4ff9-a142-6ec8359fbd4c service neutron] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:12:25.811520 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1392/2785] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Aug 30 02:12:25 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 39 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 02:12:28.183370 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02a5c9b5-f95c-4203-a717-b1e6ff69fd6f None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:12:28.183828 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02a5c9b5-f95c-4203-a717-b1e6ff69fd6f None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:12:28.183928 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02a5c9b5-f95c-4203-a717-b1e6ff69fd6f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:12:28.232839 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-02a5c9b5-f95c-4203-a717-b1e6ff69fd6f None None] MFA Rules not processed for user `121c374d49a64335ab74a431cfbc1320`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:12:28.233997 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-02a5c9b5-f95c-4203-a717-b1e6ff69fd6f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:12:28.250230 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1394/2786] 10.222.0.27 () {58 vars in 1009 bytes} [Tue Aug 30 02:12:28 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 68 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 02:12:28.262250 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0571ee65-c41b-4ff9-a142-6ec8359fbd4c service neutron] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:12:28.284145 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0571ee65-c41b-4ff9-a142-6ec8359fbd4c service neutron] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:12:28.288985 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8743143f-4b61-491c-a28e-be50db8041ae service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b4f0008fde424f22bc1f7307cf422774', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4562f3951fbe465280073e60a11e44e2', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:12:28.289599 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8743143f-4b61-491c-a28e-be50db8041ae service swift] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:12:28.289877 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8743143f-4b61-491c-a28e-be50db8041ae service swift] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:12:28.290143 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8743143f-4b61-491c-a28e-be50db8041ae service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:12:28.298935 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8743143f-4b61-491c-a28e-be50db8041ae service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:12:28.300488 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8743143f-4b61-491c-a28e-be50db8041ae service swift] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:12:28.308614 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1393/2787] 10.222.0.27 () {60 vars in 1384 bytes} [Tue Aug 30 02:12:28 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 47 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Aug 30 02:12:38.247371 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=36 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:12:38.305513 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=51 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:12:43.510014 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78a8ed5d-b7e3-48a9-be89-3607c07addf6 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:12:43.510485 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78a8ed5d-b7e3-48a9-be89-3607c07addf6 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:12:43.510590 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78a8ed5d-b7e3-48a9-be89-3607c07addf6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:12:43.529328 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-78a8ed5d-b7e3-48a9-be89-3607c07addf6 None None] MFA Rules not processed for user `121c374d49a64335ab74a431cfbc1320`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:12:43.530556 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-78a8ed5d-b7e3-48a9-be89-3607c07addf6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:12:43.538016 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1395/2788] 10.222.0.27 () {58 vars in 967 bytes} [Tue Aug 30 02:12:43 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 29 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 02:12:43.550533 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8743143f-4b61-491c-a28e-be50db8041ae service swift] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:12:43.561994 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8743143f-4b61-491c-a28e-be50db8041ae service swift] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:12:43.566954 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3f584ba-56b7-4ed5-9753-a9afc5415b32 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b4f0008fde424f22bc1f7307cf422774', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4562f3951fbe465280073e60a11e44e2', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:12:43.567574 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3f584ba-56b7-4ed5-9753-a9afc5415b32 service swift] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:12:43.567856 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3f584ba-56b7-4ed5-9753-a9afc5415b32 service swift] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:12:43.568129 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3f584ba-56b7-4ed5-9753-a9afc5415b32 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:12:43.576137 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3f584ba-56b7-4ed5-9753-a9afc5415b32 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:12:43.577752 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3f584ba-56b7-4ed5-9753-a9afc5415b32 service swift] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:12:43.585634 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1394/2789] 10.222.0.27 () {60 vars in 1384 bytes} [Tue Aug 30 02:12:43 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 36 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Aug 30 02:12:53.524126 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=3 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:12:53.582477 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=6 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:13:04.064579 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78a8ed5d-b7e3-48a9-be89-3607c07addf6 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:13:04.080798 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78a8ed5d-b7e3-48a9-be89-3607c07addf6 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:13:04.086460 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-026ebe50-fe78-4942-9635-b5733972cb2b service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd64f2e71361644589bec8a06e6eceed1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4562f3951fbe465280073e60a11e44e2', 'project_domain_id': 'default', 'roles': ['member', 'service', 'admin', 'ResellerAdmin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:13:04.087123 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-026ebe50-fe78-4942-9635-b5733972cb2b service nova] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:13:04.087402 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-026ebe50-fe78-4942-9635-b5733972cb2b service nova] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:13:04.087934 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-026ebe50-fe78-4942-9635-b5733972cb2b service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:13:04.089960 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-026ebe50-fe78-4942-9635-b5733972cb2b service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:13:04.102748 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-026ebe50-fe78-4942-9635-b5733972cb2b service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:13:04.104445 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-026ebe50-fe78-4942-9635-b5733972cb2b service nova] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:13:04.122713 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1396/2790] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Aug 30 02:13:04 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 52 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 30 02:13:04.189032 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3f584ba-56b7-4ed5-9753-a9afc5415b32 service swift] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:13:04.200625 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3f584ba-56b7-4ed5-9753-a9afc5415b32 service swift] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:13:04.205433 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a1b5fd3-eeef-4210-b39c-eea5baa66c98 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f3ea1580c56f4814a92bbb2acf542905', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4562f3951fbe465280073e60a11e44e2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:13:04.206290 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a1b5fd3-eeef-4210-b39c-eea5baa66c98 service neutron] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:13:04.206645 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a1b5fd3-eeef-4210-b39c-eea5baa66c98 service neutron] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:13:04.207011 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a1b5fd3-eeef-4210-b39c-eea5baa66c98 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:13:04.208729 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-5a1b5fd3-eeef-4210-b39c-eea5baa66c98 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:13:04.219195 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a1b5fd3-eeef-4210-b39c-eea5baa66c98 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:13:04.220685 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a1b5fd3-eeef-4210-b39c-eea5baa66c98 service neutron] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:13:04.230018 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1395/2791] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Aug 30 02:13:04 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 42 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 02:13:06.577524 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-026ebe50-fe78-4942-9635-b5733972cb2b service nova] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:13:06.588703 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-026ebe50-fe78-4942-9635-b5733972cb2b service nova] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:13:06.594978 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-536a0dcb-3224-40de-af2a-b2da1fae7162 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd64f2e71361644589bec8a06e6eceed1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4562f3951fbe465280073e60a11e44e2', 'project_domain_id': 'default', 'roles': ['member', 'service', 'admin', 'ResellerAdmin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:13:06.595795 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-536a0dcb-3224-40de-af2a-b2da1fae7162 service nova] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:13:06.596153 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-536a0dcb-3224-40de-af2a-b2da1fae7162 service nova] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:13:06.596426 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-536a0dcb-3224-40de-af2a-b2da1fae7162 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:13:06.598173 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-536a0dcb-3224-40de-af2a-b2da1fae7162 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:13:06.611901 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-536a0dcb-3224-40de-af2a-b2da1fae7162 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:13:06.613657 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-536a0dcb-3224-40de-af2a-b2da1fae7162 service nova] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:13:06.624294 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1397/2792] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Aug 30 02:13:06 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 47 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 02:13:14.226349 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=6 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:13:16.063419 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a1b5fd3-eeef-4210-b39c-eea5baa66c98 service neutron] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:13:16.069159 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-5a1b5fd3-eeef-4210-b39c-eea5baa66c98 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:13:16.163121 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a1b5fd3-eeef-4210-b39c-eea5baa66c98 service neutron] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:13:16.165039 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-adcc868f-4a25-4b7e-8408-41657636998e service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ab936b3767a8417aa80c8c8f1b84b667', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4562f3951fbe465280073e60a11e44e2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:13:16.165893 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adcc868f-4a25-4b7e-8408-41657636998e service cinder] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:13:16.166170 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adcc868f-4a25-4b7e-8408-41657636998e service cinder] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:13:16.166439 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adcc868f-4a25-4b7e-8408-41657636998e service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:13:16.168206 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-adcc868f-4a25-4b7e-8408-41657636998e service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:13:16.242561 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-adcc868f-4a25-4b7e-8408-41657636998e service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:13:16.243981 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-adcc868f-4a25-4b7e-8408-41657636998e service cinder] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:13:16.254664 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1396/2793] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Aug 30 02:13:16 2022] GET /identity/v3/auth/tokens => generated 3266 bytes in 192 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 02:13:16.619850 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=12 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:13:16.676025 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-536a0dcb-3224-40de-af2a-b2da1fae7162 service nova] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:13:16.677560 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-536a0dcb-3224-40de-af2a-b2da1fae7162 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:13:16.749099 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-536a0dcb-3224-40de-af2a-b2da1fae7162 service nova] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:13:16.753735 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3eb1518d-9eab-4230-8028-74130e80af25 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:13:16.754364 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3eb1518d-9eab-4230-8028-74130e80af25 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:13:16.754653 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3eb1518d-9eab-4230-8028-74130e80af25 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:13:16.754912 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3eb1518d-9eab-4230-8028-74130e80af25 admin admin] PATH_INFO: `/v3/users/04f1ec1a3d964e34838b14f636cdf13d` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:13:16.756040 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3eb1518d-9eab-4230-8028-74130e80af25 admin admin] RBAC: Authorizing `identity:delete_user(user_id=04f1ec1a3d964e34838b14f636cdf13d)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:13:16.757971 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3eb1518d-9eab-4230-8028-74130e80af25 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:13:16.909315 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-3eb1518d-9eab-4230-8028-74130e80af25 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '04f1ec1a3d964e34838b14f636cdf13d'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:13:16.910006 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-3eb1518d-9eab-4230-8028-74130e80af25 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '04f1ec1a3d964e34838b14f636cdf13d'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:13:16.910618 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-3eb1518d-9eab-4230-8028-74130e80af25 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '04f1ec1a3d964e34838b14f636cdf13d'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:13:16.923287 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-3eb1518d-9eab-4230-8028-74130e80af25 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '04f1ec1a3d964e34838b14f636cdf13d'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:13:16.930438 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-3eb1518d-9eab-4230-8028-74130e80af25 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '04f1ec1a3d964e34838b14f636cdf13d'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:13:17.003837 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1398/2794] 10.222.0.27 () {58 vars in 1224 bytes} [Tue Aug 30 02:13:16 2022] DELETE /identity/v3/users/04f1ec1a3d964e34838b14f636cdf13d => generated 0 bytes in 328 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 02:13:17.014298 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-adcc868f-4a25-4b7e-8408-41657636998e service cinder] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:13:17.023690 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-adcc868f-4a25-4b7e-8408-41657636998e service cinder] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:13:17.072237 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f85bb48f-c825-4053-8f16-77ecf2f68e1b service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f3ea1580c56f4814a92bbb2acf542905', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4562f3951fbe465280073e60a11e44e2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:13:17.072894 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f85bb48f-c825-4053-8f16-77ecf2f68e1b service neutron] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:13:17.073100 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f85bb48f-c825-4053-8f16-77ecf2f68e1b service neutron] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:13:17.073367 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f85bb48f-c825-4053-8f16-77ecf2f68e1b service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:13:17.075920 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-f85bb48f-c825-4053-8f16-77ecf2f68e1b service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:13:17.138139 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f85bb48f-c825-4053-8f16-77ecf2f68e1b service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:13:17.139717 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f85bb48f-c825-4053-8f16-77ecf2f68e1b service neutron] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:13:17.150135 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1397/2795] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Aug 30 02:13:17 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 136 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 30 02:13:17.365394 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3eb1518d-9eab-4230-8028-74130e80af25 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:13:17.366887 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-3eb1518d-9eab-4230-8028-74130e80af25 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:13:17.381633 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3eb1518d-9eab-4230-8028-74130e80af25 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:13:17.385933 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f6d3430-9113-407f-8f04-0bb595f38172 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:13:17.386595 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f6d3430-9113-407f-8f04-0bb595f38172 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:13:17.386874 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f6d3430-9113-407f-8f04-0bb595f38172 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:13:17.387144 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f6d3430-9113-407f-8f04-0bb595f38172 admin admin] PATH_INFO: `/v3/projects/7e5b69be3a9b4ec0b5038f51a918bf27` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:13:17.388814 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f6d3430-9113-407f-8f04-0bb595f38172 admin admin] RBAC: Authorizing `identity:delete_project(project_id=7e5b69be3a9b4ec0b5038f51a918bf27)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:13:17.390724 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f6d3430-9113-407f-8f04-0bb595f38172 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:13:17.471989 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-7f6d3430-9113-407f-8f04-0bb595f38172 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '7e5b69be3a9b4ec0b5038f51a918bf27', 'request_id': 'req-7f6d3430-9113-407f-8f04-0bb595f38172'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:13:17.477480 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-7f6d3430-9113-407f-8f04-0bb595f38172 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '7e5b69be3a9b4ec0b5038f51a918bf27', 'request_id': 'req-7f6d3430-9113-407f-8f04-0bb595f38172'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:13:17.484485 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-7f6d3430-9113-407f-8f04-0bb595f38172 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '7e5b69be3a9b4ec0b5038f51a918bf27', 'request_id': 'req-7f6d3430-9113-407f-8f04-0bb595f38172'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:13:17.528046 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-7f6d3430-9113-407f-8f04-0bb595f38172 admin admin] The token cache is being invalidate because project 7e5b69be3a9b4ec0b5038f51a918bf27 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81577) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 30 02:13:17.528362 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-7f6d3430-9113-407f-8f04-0bb595f38172 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:13:17.529003 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-7f6d3430-9113-407f-8f04-0bb595f38172 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:13:17.530620 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1399/2796] 10.222.0.27 () {58 vars in 1233 bytes} [Tue Aug 30 02:13:17 2022] DELETE /identity/v3/projects/7e5b69be3a9b4ec0b5038f51a918bf27 => generated 0 bytes in 166 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 02:13:17.535169 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f85bb48f-c825-4053-8f16-77ecf2f68e1b service neutron] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:13:17.540407 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=100 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:13:17.544389 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f85bb48f-c825-4053-8f16-77ecf2f68e1b service neutron] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:13:17.600766 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6230c0fa-a0a1-46b6-9f6f-709e46e567df admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:13:17.601394 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6230c0fa-a0a1-46b6-9f6f-709e46e567df admin admin] REQUEST_METHOD: `DELETE` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:13:17.601695 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6230c0fa-a0a1-46b6-9f6f-709e46e567df admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:13:17.601963 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6230c0fa-a0a1-46b6-9f6f-709e46e567df admin admin] PATH_INFO: `/v3/users/8d5dd61614e242a3b7c1a28a2627de8c` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:13:17.603056 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6230c0fa-a0a1-46b6-9f6f-709e46e567df admin admin] RBAC: Authorizing `identity:delete_user(user_id=8d5dd61614e242a3b7c1a28a2627de8c)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:13:17.604930 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6230c0fa-a0a1-46b6-9f6f-709e46e567df admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:13:17.782324 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-6230c0fa-a0a1-46b6-9f6f-709e46e567df admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '8d5dd61614e242a3b7c1a28a2627de8c'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:13:17.782899 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-6230c0fa-a0a1-46b6-9f6f-709e46e567df admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '8d5dd61614e242a3b7c1a28a2627de8c'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:13:17.854632 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-6230c0fa-a0a1-46b6-9f6f-709e46e567df admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '8d5dd61614e242a3b7c1a28a2627de8c'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:13:17.855417 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-6230c0fa-a0a1-46b6-9f6f-709e46e567df admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '8d5dd61614e242a3b7c1a28a2627de8c'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:13:17.864154 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-6230c0fa-a0a1-46b6-9f6f-709e46e567df admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '8d5dd61614e242a3b7c1a28a2627de8c'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:13:17.882978 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1398/2797] 10.222.0.27 () {58 vars in 1224 bytes} [Tue Aug 30 02:13:17 2022] DELETE /identity/v3/users/8d5dd61614e242a3b7c1a28a2627de8c => generated 0 bytes in 349 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 02:13:18.125727 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f6d3430-9113-407f-8f04-0bb595f38172 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:13:18.127218 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-7f6d3430-9113-407f-8f04-0bb595f38172 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:13:18.135303 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=81,keystone:DELETE=17,keystone:INSERT=2 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:13:18.147199 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f6d3430-9113-407f-8f04-0bb595f38172 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:13:18.205676 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33fb4689-0bd6-4194-afab-7fe8aa2a6525 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:13:18.206326 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33fb4689-0bd6-4194-afab-7fe8aa2a6525 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:13:18.206618 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33fb4689-0bd6-4194-afab-7fe8aa2a6525 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:13:18.206909 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33fb4689-0bd6-4194-afab-7fe8aa2a6525 admin admin] PATH_INFO: `/v3/projects/77e669ae5c964911b583ab638626d846` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:13:18.208071 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33fb4689-0bd6-4194-afab-7fe8aa2a6525 admin admin] RBAC: Authorizing `identity:delete_project(project_id=77e669ae5c964911b583ab638626d846)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:13:18.210059 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33fb4689-0bd6-4194-afab-7fe8aa2a6525 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:13:18.318603 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-33fb4689-0bd6-4194-afab-7fe8aa2a6525 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '77e669ae5c964911b583ab638626d846', 'request_id': 'req-33fb4689-0bd6-4194-afab-7fe8aa2a6525'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:13:18.324648 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-33fb4689-0bd6-4194-afab-7fe8aa2a6525 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '77e669ae5c964911b583ab638626d846', 'request_id': 'req-33fb4689-0bd6-4194-afab-7fe8aa2a6525'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:13:18.331716 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-33fb4689-0bd6-4194-afab-7fe8aa2a6525 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '77e669ae5c964911b583ab638626d846', 'request_id': 'req-33fb4689-0bd6-4194-afab-7fe8aa2a6525'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:13:18.407084 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-33fb4689-0bd6-4194-afab-7fe8aa2a6525 admin admin] The token cache is being invalidate because project 77e669ae5c964911b583ab638626d846 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81577) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 30 02:13:18.410133 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-33fb4689-0bd6-4194-afab-7fe8aa2a6525 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:13:18.410133 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-33fb4689-0bd6-4194-afab-7fe8aa2a6525 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:13:18.410805 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1400/2798] 10.222.0.27 () {58 vars in 1233 bytes} [Tue Aug 30 02:13:18 2022] DELETE /identity/v3/projects/77e669ae5c964911b583ab638626d846 => generated 0 bytes in 284 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 02:13:24.856844 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6230c0fa-a0a1-46b6-9f6f-709e46e567df admin admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:13:24.858549 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-6230c0fa-a0a1-46b6-9f6f-709e46e567df admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:13:24.914700 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6230c0fa-a0a1-46b6-9f6f-709e46e567df admin admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:13:24.951428 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:13:24.961778 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-505c463e-df7a-4612-8eab-85a09d3808c8 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f3ea1580c56f4814a92bbb2acf542905', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4562f3951fbe465280073e60a11e44e2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:13:24.962379 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-505c463e-df7a-4612-8eab-85a09d3808c8 service neutron] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:13:24.962655 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-505c463e-df7a-4612-8eab-85a09d3808c8 service neutron] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:13:24.962921 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-505c463e-df7a-4612-8eab-85a09d3808c8 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:13:24.964623 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-505c463e-df7a-4612-8eab-85a09d3808c8 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:13:25.036602 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-505c463e-df7a-4612-8eab-85a09d3808c8 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:13:25.037974 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-505c463e-df7a-4612-8eab-85a09d3808c8 service neutron] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:13:25.048299 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1399/2799] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Aug 30 02:13:24 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 192 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 02:13:28.403793 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=50,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:13:28.680499 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33fb4689-0bd6-4194-afab-7fe8aa2a6525 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:13:28.682098 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-33fb4689-0bd6-4194-afab-7fe8aa2a6525 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:13:28.696651 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33fb4689-0bd6-4194-afab-7fe8aa2a6525 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:13:28.702204 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa3ad5b7-2b91-46c9-905a-3101c7e350fa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:13:28.702909 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa3ad5b7-2b91-46c9-905a-3101c7e350fa admin admin] REQUEST_METHOD: `DELETE` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:13:28.703189 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa3ad5b7-2b91-46c9-905a-3101c7e350fa admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:13:28.703459 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa3ad5b7-2b91-46c9-905a-3101c7e350fa admin admin] PATH_INFO: `/v3/users/3eb33b1b787447daa68b651f9686574a` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:13:28.704601 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa3ad5b7-2b91-46c9-905a-3101c7e350fa admin admin] RBAC: Authorizing `identity:delete_user(user_id=3eb33b1b787447daa68b651f9686574a)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:13:28.706574 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa3ad5b7-2b91-46c9-905a-3101c7e350fa admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:13:28.885434 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-aa3ad5b7-2b91-46c9-905a-3101c7e350fa admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '3eb33b1b787447daa68b651f9686574a'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:13:28.886118 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-aa3ad5b7-2b91-46c9-905a-3101c7e350fa admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '3eb33b1b787447daa68b651f9686574a'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:13:28.886735 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-aa3ad5b7-2b91-46c9-905a-3101c7e350fa admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '3eb33b1b787447daa68b651f9686574a'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:13:28.898270 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-aa3ad5b7-2b91-46c9-905a-3101c7e350fa admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '3eb33b1b787447daa68b651f9686574a'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:13:28.905771 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-aa3ad5b7-2b91-46c9-905a-3101c7e350fa admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '3eb33b1b787447daa68b651f9686574a'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:13:28.991509 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1401/2800] 10.222.0.27 () {58 vars in 1224 bytes} [Tue Aug 30 02:13:28 2022] DELETE /identity/v3/users/3eb33b1b787447daa68b651f9686574a => generated 0 bytes in 312 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 02:13:29.273347 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-505c463e-df7a-4612-8eab-85a09d3808c8 service neutron] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:13:29.284213 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-505c463e-df7a-4612-8eab-85a09d3808c8 service neutron] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:13:29.341900 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67347231-f3c2-4974-ae97-c2e85e5f17dd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:13:29.342516 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67347231-f3c2-4974-ae97-c2e85e5f17dd admin admin] REQUEST_METHOD: `DELETE` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:13:29.342793 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67347231-f3c2-4974-ae97-c2e85e5f17dd admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:13:29.343063 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67347231-f3c2-4974-ae97-c2e85e5f17dd admin admin] PATH_INFO: `/v3/projects/abe39137b275484ea6dafbed8d826eb3` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:13:29.344171 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67347231-f3c2-4974-ae97-c2e85e5f17dd admin admin] RBAC: Authorizing `identity:delete_project(project_id=abe39137b275484ea6dafbed8d826eb3)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:13:29.346121 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67347231-f3c2-4974-ae97-c2e85e5f17dd admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:13:29.448110 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-67347231-f3c2-4974-ae97-c2e85e5f17dd admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'abe39137b275484ea6dafbed8d826eb3', 'request_id': 'req-67347231-f3c2-4974-ae97-c2e85e5f17dd'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:13:29.454259 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-67347231-f3c2-4974-ae97-c2e85e5f17dd admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'abe39137b275484ea6dafbed8d826eb3', 'request_id': 'req-67347231-f3c2-4974-ae97-c2e85e5f17dd'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:13:29.460316 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-67347231-f3c2-4974-ae97-c2e85e5f17dd admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'abe39137b275484ea6dafbed8d826eb3', 'request_id': 'req-67347231-f3c2-4974-ae97-c2e85e5f17dd'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:13:29.542846 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-67347231-f3c2-4974-ae97-c2e85e5f17dd admin admin] The token cache is being invalidate because project abe39137b275484ea6dafbed8d826eb3 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81578) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 30 02:13:29.543091 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-67347231-f3c2-4974-ae97-c2e85e5f17dd admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:13:29.543769 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-67347231-f3c2-4974-ae97-c2e85e5f17dd admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:13:29.545247 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1400/2801] 10.222.0.27 () {58 vars in 1233 bytes} [Tue Aug 30 02:13:29 2022] DELETE /identity/v3/projects/abe39137b275484ea6dafbed8d826eb3 => generated 0 bytes in 272 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 02:13:29.550288 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2599a234-14e4-4ad7-b818-15dc20979022 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:13:29.551032 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2599a234-14e4-4ad7-b818-15dc20979022 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:13:29.551230 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2599a234-14e4-4ad7-b818-15dc20979022 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:13:29.571218 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-2599a234-14e4-4ad7-b818-15dc20979022 None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:13:29.572417 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-2599a234-14e4-4ad7-b818-15dc20979022 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:13:29.630233 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1402/2802] 10.222.0.27 () {58 vars in 958 bytes} [Tue Aug 30 02:13:29 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 81 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 02:13:29.638015 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a618e276-7797-4a30-9b41-4fc54705f9bf None None] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:13:29.639090 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a618e276-7797-4a30-9b41-4fc54705f9bf None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:13:29.639448 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a618e276-7797-4a30-9b41-4fc54705f9bf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:13:29.659906 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.auth.core [None req-a618e276-7797-4a30-9b41-4fc54705f9bf None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81578) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:13:29.661108 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-a618e276-7797-4a30-9b41-4fc54705f9bf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:13:29.669761 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1401/2803] 10.222.0.27 () {58 vars in 958 bytes} [Tue Aug 30 02:13:29 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 02:13:29.673910 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2599a234-14e4-4ad7-b818-15dc20979022 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:13:29.683547 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2599a234-14e4-4ad7-b818-15dc20979022 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:13:29.688206 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68510626-aca1-4193-af98-aa2e0982f3d9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:13:29.688807 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68510626-aca1-4193-af98-aa2e0982f3d9 admin admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:13:29.689088 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68510626-aca1-4193-af98-aa2e0982f3d9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:13:29.689358 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68510626-aca1-4193-af98-aa2e0982f3d9 admin admin] PATH_INFO: `/v3/domains` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:13:29.690059 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68510626-aca1-4193-af98-aa2e0982f3d9 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 02:13:29.690363 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68510626-aca1-4193-af98-aa2e0982f3d9 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:13:29.691973 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68510626-aca1-4193-af98-aa2e0982f3d9 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:13:29.704462 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1403/2804] 10.222.0.27 () {58 vars in 1153 bytes} [Tue Aug 30 02:13:29 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 02:13:29.708950 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a618e276-7797-4a30-9b41-4fc54705f9bf None None] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:13:29.719093 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a618e276-7797-4a30-9b41-4fc54705f9bf None None] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:13:29.723942 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd8ec36e-b233-4f21-b41e-e842e99fa008 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:13:29.724544 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd8ec36e-b233-4f21-b41e-e842e99fa008 admin admin] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:13:29.724840 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd8ec36e-b233-4f21-b41e-e842e99fa008 admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:13:29.725095 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd8ec36e-b233-4f21-b41e-e842e99fa008 admin admin] PATH_INFO: `/v3/projects` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:13:29.725762 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd8ec36e-b233-4f21-b41e-e842e99fa008 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:13:29.727641 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd8ec36e-b233-4f21-b41e-e842e99fa008 admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:13:29.740171 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:13:29.824607 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1402/2805] 10.222.0.27 () {60 vars in 1153 bytes} [Tue Aug 30 02:13:29 2022] POST /identity/v3/projects => generated 363 bytes in 116 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 30 02:13:29.828519 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68510626-aca1-4193-af98-aa2e0982f3d9 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:13:29.838394 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68510626-aca1-4193-af98-aa2e0982f3d9 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:13:29.897350 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66930d5f-7afb-4060-866e-3275cb53e7b1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:13:29.898109 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66930d5f-7afb-4060-866e-3275cb53e7b1 admin admin] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:13:29.898448 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66930d5f-7afb-4060-866e-3275cb53e7b1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:13:29.898800 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66930d5f-7afb-4060-866e-3275cb53e7b1 admin admin] PATH_INFO: `/v3/users` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:13:29.899491 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66930d5f-7afb-4060-866e-3275cb53e7b1 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:13:29.901451 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66930d5f-7afb-4060-866e-3275cb53e7b1 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:13:29.910224 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:13:30.036106 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1404/2806] 10.222.0.27 () {60 vars in 1144 bytes} [Tue Aug 30 02:13:29 2022] POST /identity/v3/users => generated 282 bytes in 208 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 02:13:30.040218 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd8ec36e-b233-4f21-b41e-e842e99fa008 admin admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:13:30.050200 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd8ec36e-b233-4f21-b41e-e842e99fa008 admin admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:13:30.054650 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c480524a-5da0-4756-9a6b-8fe52bd98329 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:13:30.055240 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c480524a-5da0-4756-9a6b-8fe52bd98329 admin admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:13:30.055516 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c480524a-5da0-4756-9a6b-8fe52bd98329 admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:13:30.055786 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c480524a-5da0-4756-9a6b-8fe52bd98329 admin admin] PATH_INFO: `/v3/roles` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:13:30.056419 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c480524a-5da0-4756-9a6b-8fe52bd98329 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:13:30.058040 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c480524a-5da0-4756-9a6b-8fe52bd98329 admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:13:30.068082 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1403/2807] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Aug 30 02:13:30 2022] GET /identity/v3/roles => generated 1420 bytes in 28 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 02:13:30.072081 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66930d5f-7afb-4060-866e-3275cb53e7b1 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:13:30.083351 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66930d5f-7afb-4060-866e-3275cb53e7b1 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:13:30.088078 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fb47794-1094-440d-bb36-0b6982a91ae7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:13:30.088665 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fb47794-1094-440d-bb36-0b6982a91ae7 admin admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:13:30.088942 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fb47794-1094-440d-bb36-0b6982a91ae7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:13:30.089210 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fb47794-1094-440d-bb36-0b6982a91ae7 admin admin] PATH_INFO: `/v3/roles` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:13:30.089883 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fb47794-1094-440d-bb36-0b6982a91ae7 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:13:30.092208 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fb47794-1094-440d-bb36-0b6982a91ae7 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:13:30.102700 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1405/2808] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Aug 30 02:13:30 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 02:13:30.106890 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c480524a-5da0-4756-9a6b-8fe52bd98329 admin admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:13:30.117174 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c480524a-5da0-4756-9a6b-8fe52bd98329 admin admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:13:30.121906 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc9113ee-479d-4198-bcf7-c0c38709acf8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:13:30.122529 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc9113ee-479d-4198-bcf7-c0c38709acf8 admin admin] REQUEST_METHOD: `PUT` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:13:30.123593 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc9113ee-479d-4198-bcf7-c0c38709acf8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:13:30.123869 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc9113ee-479d-4198-bcf7-c0c38709acf8 admin admin] PATH_INFO: `/v3/projects/34221060897e45e4a1e8f81af791c8bf/users/42589196bd6f479ab4f13d0c7e152200/roles/1fb81afcb9f245f3b989add94598a063` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:13:30.139566 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc9113ee-479d-4198-bcf7-c0c38709acf8 admin admin] RBAC: Authorizing `identity:create_grant(project_id=34221060897e45e4a1e8f81af791c8bf, user_id=42589196bd6f479ab4f13d0c7e152200, role_id=1fb81afcb9f245f3b989add94598a063)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:13:30.142259 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc9113ee-479d-4198-bcf7-c0c38709acf8 admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:13:30.203761 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1404/2809] 10.222.0.27 () {60 vars in 1483 bytes} [Tue Aug 30 02:13:30 2022] PUT /identity/v3/projects/34221060897e45e4a1e8f81af791c8bf/users/42589196bd6f479ab4f13d0c7e152200/roles/1fb81afcb9f245f3b989add94598a063 => generated 0 bytes in 97 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 02:13:30.215071 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b893ab98-0e67-43c3-aac0-b9bec665185f None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:13:30.216183 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b893ab98-0e67-43c3-aac0-b9bec665185f None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:13:30.216508 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b893ab98-0e67-43c3-aac0-b9bec665185f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:13:30.243733 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-b893ab98-0e67-43c3-aac0-b9bec665185f None None] MFA Rules not processed for user `42589196bd6f479ab4f13d0c7e152200`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:13:30.244931 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-b893ab98-0e67-43c3-aac0-b9bec665185f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:13:30.308372 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1406/2810] 10.222.0.27 () {58 vars in 958 bytes} [Tue Aug 30 02:13:30 2022] POST /identity/v3/auth/tokens => generated 3240 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 02:13:30.313502 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc9113ee-479d-4198-bcf7-c0c38709acf8 admin admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:13:30.375331 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc9113ee-479d-4198-bcf7-c0c38709acf8 admin admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:13:30.379799 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6606da7-8103-4c44-a35f-f4fdcd240762 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:13:30.380392 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6606da7-8103-4c44-a35f-f4fdcd240762 admin admin] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:13:30.380693 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6606da7-8103-4c44-a35f-f4fdcd240762 admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:13:30.380959 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6606da7-8103-4c44-a35f-f4fdcd240762 admin admin] PATH_INFO: `/v3/projects` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:13:30.381612 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6606da7-8103-4c44-a35f-f4fdcd240762 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:13:30.383480 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6606da7-8103-4c44-a35f-f4fdcd240762 admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:13:30.485209 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1405/2811] 10.222.0.27 () {60 vars in 1153 bytes} [Tue Aug 30 02:13:30 2022] POST /identity/v3/projects => generated 365 bytes in 172 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 02:13:30.489287 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b893ab98-0e67-43c3-aac0-b9bec665185f None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:13:30.500353 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b893ab98-0e67-43c3-aac0-b9bec665185f None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:13:30.556652 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-777ce576-fc29-46dd-b103-62f775e40049 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:13:30.557304 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-777ce576-fc29-46dd-b103-62f775e40049 admin admin] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:13:30.557547 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-777ce576-fc29-46dd-b103-62f775e40049 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:13:30.557821 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-777ce576-fc29-46dd-b103-62f775e40049 admin admin] PATH_INFO: `/v3/users` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:13:30.558446 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-777ce576-fc29-46dd-b103-62f775e40049 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:13:30.560394 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-777ce576-fc29-46dd-b103-62f775e40049 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:13:30.619891 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1407/2812] 10.222.0.27 () {60 vars in 1144 bytes} [Tue Aug 30 02:13:30 2022] POST /identity/v3/users => generated 289 bytes in 131 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 02:13:30.624271 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6606da7-8103-4c44-a35f-f4fdcd240762 admin admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:13:30.634586 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6606da7-8103-4c44-a35f-f4fdcd240762 admin admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:13:30.639908 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed3f1641-6eff-49fc-8825-f60044790676 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:13:30.640490 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed3f1641-6eff-49fc-8825-f60044790676 admin admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:13:30.640765 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed3f1641-6eff-49fc-8825-f60044790676 admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:13:30.641032 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed3f1641-6eff-49fc-8825-f60044790676 admin admin] PATH_INFO: `/v3/roles` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:13:30.641685 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed3f1641-6eff-49fc-8825-f60044790676 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:13:30.643256 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed3f1641-6eff-49fc-8825-f60044790676 admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:13:30.654479 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1406/2813] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Aug 30 02:13:30 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 02:13:30.658425 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-777ce576-fc29-46dd-b103-62f775e40049 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:13:30.668246 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-777ce576-fc29-46dd-b103-62f775e40049 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:13:30.673149 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57f5f74e-b5d5-408f-a550-03e6c6869764 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:13:30.673818 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57f5f74e-b5d5-408f-a550-03e6c6869764 admin admin] REQUEST_METHOD: `PUT` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:13:30.674103 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57f5f74e-b5d5-408f-a550-03e6c6869764 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:13:30.674371 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57f5f74e-b5d5-408f-a550-03e6c6869764 admin admin] PATH_INFO: `/v3/projects/1a78493e47ba4b5d94fd55ab4a5ec349/users/6b7d792abd08454aa39bce290ed505d0/roles/ead5fac86a0747ea8823a82efcea8ed7` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:13:30.689504 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57f5f74e-b5d5-408f-a550-03e6c6869764 admin admin] RBAC: Authorizing `identity:create_grant(project_id=1a78493e47ba4b5d94fd55ab4a5ec349, user_id=6b7d792abd08454aa39bce290ed505d0, role_id=ead5fac86a0747ea8823a82efcea8ed7)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:13:30.692807 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57f5f74e-b5d5-408f-a550-03e6c6869764 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:13:30.764289 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1408/2814] 10.222.0.27 () {60 vars in 1483 bytes} [Tue Aug 30 02:13:30 2022] PUT /identity/v3/projects/1a78493e47ba4b5d94fd55ab4a5ec349/users/6b7d792abd08454aa39bce290ed505d0/roles/ead5fac86a0747ea8823a82efcea8ed7 => generated 0 bytes in 106 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 02:13:30.768341 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed3f1641-6eff-49fc-8825-f60044790676 admin admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:13:30.779262 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed3f1641-6eff-49fc-8825-f60044790676 admin admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:13:30.836327 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d350c966-7cce-49cd-83de-7423513b6ae4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:13:30.837057 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d350c966-7cce-49cd-83de-7423513b6ae4 admin admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:13:30.837331 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d350c966-7cce-49cd-83de-7423513b6ae4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:13:30.837628 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d350c966-7cce-49cd-83de-7423513b6ae4 admin admin] PATH_INFO: `/v3/roles` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:13:30.838251 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d350c966-7cce-49cd-83de-7423513b6ae4 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:13:30.839820 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d350c966-7cce-49cd-83de-7423513b6ae4 admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:13:30.849502 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1407/2815] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Aug 30 02:13:30 2022] GET /identity/v3/roles => generated 1420 bytes in 82 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 02:13:30.853626 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57f5f74e-b5d5-408f-a550-03e6c6869764 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:13:30.864510 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57f5f74e-b5d5-408f-a550-03e6c6869764 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:13:30.869009 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38615bbe-7408-4127-9d10-87a385d3deaa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:13:30.869686 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38615bbe-7408-4127-9d10-87a385d3deaa admin admin] REQUEST_METHOD: `PUT` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:13:30.869959 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38615bbe-7408-4127-9d10-87a385d3deaa admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:13:30.870223 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38615bbe-7408-4127-9d10-87a385d3deaa admin admin] PATH_INFO: `/v3/projects/1a78493e47ba4b5d94fd55ab4a5ec349/users/6b7d792abd08454aa39bce290ed505d0/roles/ead5fac86a0747ea8823a82efcea8ed7` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:13:30.872807 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38615bbe-7408-4127-9d10-87a385d3deaa admin admin] RBAC: Authorizing `identity:create_grant(project_id=1a78493e47ba4b5d94fd55ab4a5ec349, user_id=6b7d792abd08454aa39bce290ed505d0, role_id=ead5fac86a0747ea8823a82efcea8ed7)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:13:30.875466 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38615bbe-7408-4127-9d10-87a385d3deaa admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:13:30.885524 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1409/2816] 10.222.0.27 () {60 vars in 1483 bytes} [Tue Aug 30 02:13:30 2022] PUT /identity/v3/projects/1a78493e47ba4b5d94fd55ab4a5ec349/users/6b7d792abd08454aa39bce290ed505d0/roles/ead5fac86a0747ea8823a82efcea8ed7 => generated 0 bytes in 32 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 02:13:30.895503 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74587179-f5b2-4f62-b4bf-ec2b13b88b1b None None] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:13:30.895873 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74587179-f5b2-4f62-b4bf-ec2b13b88b1b None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:13:30.896150 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74587179-f5b2-4f62-b4bf-ec2b13b88b1b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:13:30.924495 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.auth.core [None req-74587179-f5b2-4f62-b4bf-ec2b13b88b1b None None] MFA Rules not processed for user `6b7d792abd08454aa39bce290ed505d0`. Rule list: `[]` (Enabled: `True`). {{(pid=81578) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:13:30.925740 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-74587179-f5b2-4f62-b4bf-ec2b13b88b1b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:13:30.930643 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:13:30.993552 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1408/2817] 10.222.0.27 () {58 vars in 958 bytes} [Tue Aug 30 02:13:30 2022] POST /identity/v3/auth/tokens => generated 3309 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 30 02:13:31.005879 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38615bbe-7408-4127-9d10-87a385d3deaa admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:13:31.007625 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-38615bbe-7408-4127-9d10-87a385d3deaa admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:13:31.029973 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:INSERT=8,keystone:SELECT=92 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:13:31.065093 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38615bbe-7408-4127-9d10-87a385d3deaa admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:13:31.068706 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a4888f5-47a3-405b-9927-872c52675907 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ab936b3767a8417aa80c8c8f1b84b667', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4562f3951fbe465280073e60a11e44e2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:13:31.069296 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a4888f5-47a3-405b-9927-872c52675907 service cinder] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:13:31.069605 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a4888f5-47a3-405b-9927-872c52675907 service cinder] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:13:31.069874 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a4888f5-47a3-405b-9927-872c52675907 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:13:31.078661 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a4888f5-47a3-405b-9927-872c52675907 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:13:31.080061 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a4888f5-47a3-405b-9927-872c52675907 service cinder] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:13:31.089137 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1410/2818] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Aug 30 02:13:31 2022] GET /identity/v3/auth/tokens => generated 3309 bytes in 84 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 02:13:31.966365 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74587179-f5b2-4f62-b4bf-ec2b13b88b1b None None] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:13:32.028941 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74587179-f5b2-4f62-b4bf-ec2b13b88b1b None None] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:13:32.033390 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28f8f6cd-6413-4fe7-9e89-b617e54e254b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:13:32.034043 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28f8f6cd-6413-4fe7-9e89-b617e54e254b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:13:32.034317 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28f8f6cd-6413-4fe7-9e89-b617e54e254b admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:13:32.034586 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28f8f6cd-6413-4fe7-9e89-b617e54e254b admin admin] PATH_INFO: `/v3/users/42589196bd6f479ab4f13d0c7e152200` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:13:32.035680 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28f8f6cd-6413-4fe7-9e89-b617e54e254b admin admin] RBAC: Authorizing `identity:delete_user(user_id=42589196bd6f479ab4f13d0c7e152200)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:13:32.037694 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28f8f6cd-6413-4fe7-9e89-b617e54e254b admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:13:32.187543 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-28f8f6cd-6413-4fe7-9e89-b617e54e254b admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '42589196bd6f479ab4f13d0c7e152200'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:13:32.188314 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-28f8f6cd-6413-4fe7-9e89-b617e54e254b admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '42589196bd6f479ab4f13d0c7e152200'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:13:32.246638 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-28f8f6cd-6413-4fe7-9e89-b617e54e254b admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '42589196bd6f479ab4f13d0c7e152200'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:13:32.247229 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-28f8f6cd-6413-4fe7-9e89-b617e54e254b admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '42589196bd6f479ab4f13d0c7e152200'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:13:32.254489 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-28f8f6cd-6413-4fe7-9e89-b617e54e254b admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '42589196bd6f479ab4f13d0c7e152200'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:13:32.272724 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1409/2819] 10.222.0.27 () {58 vars in 1224 bytes} [Tue Aug 30 02:13:31 2022] DELETE /identity/v3/users/42589196bd6f479ab4f13d0c7e152200 => generated 0 bytes in 307 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 02:13:32.277218 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b57cfe8-d39c-4716-b137-c1257b718e89 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:13:32.278348 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b57cfe8-d39c-4716-b137-c1257b718e89 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:13:32.278447 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b57cfe8-d39c-4716-b137-c1257b718e89 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:13:32.296890 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-1b57cfe8-d39c-4716-b137-c1257b718e89 None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:13:32.298169 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-1b57cfe8-d39c-4716-b137-c1257b718e89 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:13:32.357376 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1411/2820] 10.222.0.27 () {58 vars in 958 bytes} [Tue Aug 30 02:13:32 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 81 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 30 02:13:32.367380 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28f8f6cd-6413-4fe7-9e89-b617e54e254b admin admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:13:32.368889 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-28f8f6cd-6413-4fe7-9e89-b617e54e254b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:13:32.398882 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:13:32.430245 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28f8f6cd-6413-4fe7-9e89-b617e54e254b admin admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:13:32.433927 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de0c7aa5-375b-419f-bd13-3bd7df066b8a service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f3ea1580c56f4814a92bbb2acf542905', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4562f3951fbe465280073e60a11e44e2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:13:32.435285 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de0c7aa5-375b-419f-bd13-3bd7df066b8a service neutron] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:13:32.435285 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de0c7aa5-375b-419f-bd13-3bd7df066b8a service neutron] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:13:32.435285 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de0c7aa5-375b-419f-bd13-3bd7df066b8a service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:13:32.444137 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de0c7aa5-375b-419f-bd13-3bd7df066b8a service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:13:32.445578 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de0c7aa5-375b-419f-bd13-3bd7df066b8a service neutron] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:13:32.455701 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1410/2821] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Aug 30 02:13:32 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 89 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 02:13:32.879114 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b57cfe8-d39c-4716-b137-c1257b718e89 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:13:32.880526 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-1b57cfe8-d39c-4716-b137-c1257b718e89 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:13:32.892766 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b57cfe8-d39c-4716-b137-c1257b718e89 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:13:32.897409 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51082e2e-08f0-4a89-ba69-cba01b431f71 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:13:32.898181 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51082e2e-08f0-4a89-ba69-cba01b431f71 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:13:32.898573 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51082e2e-08f0-4a89-ba69-cba01b431f71 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:13:32.898939 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51082e2e-08f0-4a89-ba69-cba01b431f71 admin admin] PATH_INFO: `/v3/projects/34221060897e45e4a1e8f81af791c8bf` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:13:32.900135 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51082e2e-08f0-4a89-ba69-cba01b431f71 admin admin] RBAC: Authorizing `identity:delete_project(project_id=34221060897e45e4a1e8f81af791c8bf)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:13:32.902092 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51082e2e-08f0-4a89-ba69-cba01b431f71 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:13:33.009158 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-51082e2e-08f0-4a89-ba69-cba01b431f71 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '34221060897e45e4a1e8f81af791c8bf', 'request_id': 'req-51082e2e-08f0-4a89-ba69-cba01b431f71'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:13:33.014641 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-51082e2e-08f0-4a89-ba69-cba01b431f71 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '34221060897e45e4a1e8f81af791c8bf', 'request_id': 'req-51082e2e-08f0-4a89-ba69-cba01b431f71'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:13:33.020872 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-51082e2e-08f0-4a89-ba69-cba01b431f71 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '34221060897e45e4a1e8f81af791c8bf', 'request_id': 'req-51082e2e-08f0-4a89-ba69-cba01b431f71'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:13:33.112244 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-51082e2e-08f0-4a89-ba69-cba01b431f71 admin admin] The token cache is being invalidate because project 34221060897e45e4a1e8f81af791c8bf was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81577) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 30 02:13:33.112538 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-51082e2e-08f0-4a89-ba69-cba01b431f71 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:13:33.113168 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-51082e2e-08f0-4a89-ba69-cba01b431f71 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:13:33.114716 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1412/2822] 10.222.0.27 () {58 vars in 1233 bytes} [Tue Aug 30 02:13:32 2022] DELETE /identity/v3/projects/34221060897e45e4a1e8f81af791c8bf => generated 0 bytes in 236 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 02:13:33.118773 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de0c7aa5-375b-419f-bd13-3bd7df066b8a service neutron] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:13:33.128940 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de0c7aa5-375b-419f-bd13-3bd7df066b8a service neutron] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:13:33.185119 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae01f305-cf0b-4d85-937e-2d1cf8892977 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:13:33.185770 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae01f305-cf0b-4d85-937e-2d1cf8892977 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:13:33.186048 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae01f305-cf0b-4d85-937e-2d1cf8892977 admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:13:33.186315 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae01f305-cf0b-4d85-937e-2d1cf8892977 admin admin] PATH_INFO: `/v3/users/6b7d792abd08454aa39bce290ed505d0` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:13:33.187420 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae01f305-cf0b-4d85-937e-2d1cf8892977 admin admin] RBAC: Authorizing `identity:delete_user(user_id=6b7d792abd08454aa39bce290ed505d0)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:13:33.189299 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae01f305-cf0b-4d85-937e-2d1cf8892977 admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:13:33.376854 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-ae01f305-cf0b-4d85-937e-2d1cf8892977 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '6b7d792abd08454aa39bce290ed505d0'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:13:33.377545 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-ae01f305-cf0b-4d85-937e-2d1cf8892977 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '6b7d792abd08454aa39bce290ed505d0'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:13:33.443134 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-ae01f305-cf0b-4d85-937e-2d1cf8892977 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '6b7d792abd08454aa39bce290ed505d0'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:13:33.443760 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-ae01f305-cf0b-4d85-937e-2d1cf8892977 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '6b7d792abd08454aa39bce290ed505d0'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:13:33.451221 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-ae01f305-cf0b-4d85-937e-2d1cf8892977 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '6b7d792abd08454aa39bce290ed505d0'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:13:33.468812 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1411/2823] 10.222.0.27 () {58 vars in 1224 bytes} [Tue Aug 30 02:13:33 2022] DELETE /identity/v3/users/6b7d792abd08454aa39bce290ed505d0 => generated 0 bytes in 351 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 02:13:33.884705 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51082e2e-08f0-4a89-ba69-cba01b431f71 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:13:33.886376 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-51082e2e-08f0-4a89-ba69-cba01b431f71 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:13:33.902648 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51082e2e-08f0-4a89-ba69-cba01b431f71 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:13:33.947879 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:13:33.960304 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67f80f35-5b8b-4c6e-b671-4241b4ec9486 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:13:33.960935 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67f80f35-5b8b-4c6e-b671-4241b4ec9486 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:13:33.961215 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67f80f35-5b8b-4c6e-b671-4241b4ec9486 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:13:33.961516 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67f80f35-5b8b-4c6e-b671-4241b4ec9486 admin admin] PATH_INFO: `/v3/projects/1a78493e47ba4b5d94fd55ab4a5ec349` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:13:33.962625 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67f80f35-5b8b-4c6e-b671-4241b4ec9486 admin admin] RBAC: Authorizing `identity:delete_project(project_id=1a78493e47ba4b5d94fd55ab4a5ec349)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:13:33.964520 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67f80f35-5b8b-4c6e-b671-4241b4ec9486 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:13:34.115712 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-67f80f35-5b8b-4c6e-b671-4241b4ec9486 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '1a78493e47ba4b5d94fd55ab4a5ec349', 'request_id': 'req-67f80f35-5b8b-4c6e-b671-4241b4ec9486'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:13:34.121213 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-67f80f35-5b8b-4c6e-b671-4241b4ec9486 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '1a78493e47ba4b5d94fd55ab4a5ec349', 'request_id': 'req-67f80f35-5b8b-4c6e-b671-4241b4ec9486'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:13:34.128251 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-67f80f35-5b8b-4c6e-b671-4241b4ec9486 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '1a78493e47ba4b5d94fd55ab4a5ec349', 'request_id': 'req-67f80f35-5b8b-4c6e-b671-4241b4ec9486'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:13:34.228884 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-67f80f35-5b8b-4c6e-b671-4241b4ec9486 admin admin] The token cache is being invalidate because project 1a78493e47ba4b5d94fd55ab4a5ec349 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81577) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 30 02:13:34.229108 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-67f80f35-5b8b-4c6e-b671-4241b4ec9486 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:13:34.229751 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-67f80f35-5b8b-4c6e-b671-4241b4ec9486 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:13:34.231865 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1413/2824] 10.222.0.27 () {58 vars in 1233 bytes} [Tue Aug 30 02:13:33 2022] DELETE /identity/v3/projects/1a78493e47ba4b5d94fd55ab4a5ec349 => generated 0 bytes in 347 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 02:13:34.236924 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-284fb9b5-e136-4c1d-a4b9-aeeb9f6eb218 None None] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:13:34.237402 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-284fb9b5-e136-4c1d-a4b9-aeeb9f6eb218 None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:13:34.237526 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-284fb9b5-e136-4c1d-a4b9-aeeb9f6eb218 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:13:34.257993 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.auth.core [None req-284fb9b5-e136-4c1d-a4b9-aeeb9f6eb218 None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81578) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:13:34.259205 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-284fb9b5-e136-4c1d-a4b9-aeeb9f6eb218 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:13:34.299230 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:13:34.318409 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1412/2825] 10.222.0.27 () {58 vars in 958 bytes} [Tue Aug 30 02:13:34 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 82 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 02:13:34.327328 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3564761-a813-429e-87c4-3aaa764af510 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:13:34.327617 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3564761-a813-429e-87c4-3aaa764af510 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:13:34.327897 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3564761-a813-429e-87c4-3aaa764af510 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:13:34.347642 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-f3564761-a813-429e-87c4-3aaa764af510 None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:13:34.348949 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-f3564761-a813-429e-87c4-3aaa764af510 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:13:34.357529 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1414/2826] 10.222.0.27 () {58 vars in 958 bytes} [Tue Aug 30 02:13:34 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 02:13:34.362015 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-284fb9b5-e136-4c1d-a4b9-aeeb9f6eb218 None None] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:13:34.372144 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-284fb9b5-e136-4c1d-a4b9-aeeb9f6eb218 None None] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:13:34.376739 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aaaf9867-6fc0-42aa-97ae-8c7a91af4bdb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:13:34.377542 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aaaf9867-6fc0-42aa-97ae-8c7a91af4bdb admin admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:13:34.377993 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aaaf9867-6fc0-42aa-97ae-8c7a91af4bdb admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:13:34.378525 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aaaf9867-6fc0-42aa-97ae-8c7a91af4bdb admin admin] PATH_INFO: `/v3/domains` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:13:34.379168 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aaaf9867-6fc0-42aa-97ae-8c7a91af4bdb admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81578) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 02:13:34.379468 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aaaf9867-6fc0-42aa-97ae-8c7a91af4bdb admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:13:34.381072 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aaaf9867-6fc0-42aa-97ae-8c7a91af4bdb admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:13:34.392491 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1413/2827] 10.222.0.27 () {58 vars in 1153 bytes} [Tue Aug 30 02:13:34 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 30 02:13:34.396673 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3564761-a813-429e-87c4-3aaa764af510 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:13:34.406026 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3564761-a813-429e-87c4-3aaa764af510 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:13:34.410631 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-439b5401-6812-4d14-b0e4-85bb9553cf88 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:13:34.411222 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-439b5401-6812-4d14-b0e4-85bb9553cf88 admin admin] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:13:34.411502 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-439b5401-6812-4d14-b0e4-85bb9553cf88 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:13:34.411773 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-439b5401-6812-4d14-b0e4-85bb9553cf88 admin admin] PATH_INFO: `/v3/projects` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:13:34.412396 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-439b5401-6812-4d14-b0e4-85bb9553cf88 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:13:34.414367 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-439b5401-6812-4d14-b0e4-85bb9553cf88 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:13:34.499450 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1415/2828] 10.222.0.27 () {60 vars in 1153 bytes} [Tue Aug 30 02:13:34 2022] POST /identity/v3/projects => generated 375 bytes in 103 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 02:13:34.503582 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aaaf9867-6fc0-42aa-97ae-8c7a91af4bdb admin admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:13:34.513750 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aaaf9867-6fc0-42aa-97ae-8c7a91af4bdb admin admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:13:34.569227 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0d2c404-8007-4805-952c-8ca089880716 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:13:34.569856 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0d2c404-8007-4805-952c-8ca089880716 admin admin] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:13:34.570136 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0d2c404-8007-4805-952c-8ca089880716 admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:13:34.570405 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0d2c404-8007-4805-952c-8ca089880716 admin admin] PATH_INFO: `/v3/users` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:13:34.571027 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0d2c404-8007-4805-952c-8ca089880716 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:13:34.572915 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0d2c404-8007-4805-952c-8ca089880716 admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:13:34.675916 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1414/2829] 10.222.0.27 () {60 vars in 1144 bytes} [Tue Aug 30 02:13:34 2022] POST /identity/v3/users => generated 288 bytes in 173 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 02:13:34.680071 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-439b5401-6812-4d14-b0e4-85bb9553cf88 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:13:34.691710 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-439b5401-6812-4d14-b0e4-85bb9553cf88 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:13:34.696314 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-895d5490-8947-4f04-be73-6df0ed4ee6a8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:13:34.696929 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-895d5490-8947-4f04-be73-6df0ed4ee6a8 admin admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:13:34.697209 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-895d5490-8947-4f04-be73-6df0ed4ee6a8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:13:34.697501 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-895d5490-8947-4f04-be73-6df0ed4ee6a8 admin admin] PATH_INFO: `/v3/roles` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:13:34.698146 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-895d5490-8947-4f04-be73-6df0ed4ee6a8 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:13:34.699908 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-895d5490-8947-4f04-be73-6df0ed4ee6a8 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:13:34.710467 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1416/2830] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Aug 30 02:13:34 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 02:13:34.714443 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0d2c404-8007-4805-952c-8ca089880716 admin admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:13:34.727415 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0d2c404-8007-4805-952c-8ca089880716 admin admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:13:34.736077 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac8551cd-cb5d-4b6a-8dcf-890103edac55 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:13:34.736914 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac8551cd-cb5d-4b6a-8dcf-890103edac55 admin admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:13:34.737218 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac8551cd-cb5d-4b6a-8dcf-890103edac55 admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:13:34.737514 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac8551cd-cb5d-4b6a-8dcf-890103edac55 admin admin] PATH_INFO: `/v3/roles` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:13:34.738162 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac8551cd-cb5d-4b6a-8dcf-890103edac55 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:13:34.739736 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac8551cd-cb5d-4b6a-8dcf-890103edac55 admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:13:34.751399 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1415/2831] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Aug 30 02:13:34 2022] GET /identity/v3/roles => generated 1420 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 02:13:34.755514 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-895d5490-8947-4f04-be73-6df0ed4ee6a8 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:13:34.765433 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-895d5490-8947-4f04-be73-6df0ed4ee6a8 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:13:34.770046 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf1d6630-0824-4b7d-af4c-4e1fb6d817bc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:13:34.770691 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf1d6630-0824-4b7d-af4c-4e1fb6d817bc admin admin] REQUEST_METHOD: `PUT` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:13:34.770973 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf1d6630-0824-4b7d-af4c-4e1fb6d817bc admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:13:34.771245 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf1d6630-0824-4b7d-af4c-4e1fb6d817bc admin admin] PATH_INFO: `/v3/projects/885776e9f06e4b798f7fcd1c6e12339f/users/c47327e1276f447f83a96368d2d6e651/roles/1fb81afcb9f245f3b989add94598a063` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:13:34.786012 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf1d6630-0824-4b7d-af4c-4e1fb6d817bc admin admin] RBAC: Authorizing `identity:create_grant(project_id=885776e9f06e4b798f7fcd1c6e12339f, user_id=c47327e1276f447f83a96368d2d6e651, role_id=1fb81afcb9f245f3b989add94598a063)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:13:34.788619 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf1d6630-0824-4b7d-af4c-4e1fb6d817bc admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:13:34.867239 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1417/2832] 10.222.0.27 () {60 vars in 1483 bytes} [Tue Aug 30 02:13:34 2022] PUT /identity/v3/projects/885776e9f06e4b798f7fcd1c6e12339f/users/c47327e1276f447f83a96368d2d6e651/roles/1fb81afcb9f245f3b989add94598a063 => generated 0 bytes in 112 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 02:13:34.877694 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad6e0bd7-c84d-4e1e-8c42-53d24ab53c31 None None] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:13:34.878008 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad6e0bd7-c84d-4e1e-8c42-53d24ab53c31 None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:13:34.878282 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad6e0bd7-c84d-4e1e-8c42-53d24ab53c31 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:13:34.908374 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.auth.core [None req-ad6e0bd7-c84d-4e1e-8c42-53d24ab53c31 None None] MFA Rules not processed for user `c47327e1276f447f83a96368d2d6e651`. Rule list: `[]` (Enabled: `True`). {{(pid=81578) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:13:34.909714 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-ad6e0bd7-c84d-4e1e-8c42-53d24ab53c31 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:13:34.974354 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1416/2833] 10.222.0.27 () {58 vars in 958 bytes} [Tue Aug 30 02:13:34 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 02:13:34.979034 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf1d6630-0824-4b7d-af4c-4e1fb6d817bc admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:13:35.042461 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf1d6630-0824-4b7d-af4c-4e1fb6d817bc admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:13:35.047847 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de910cea-3631-4071-ad2f-0348419bdd1c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:13:35.048456 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de910cea-3631-4071-ad2f-0348419bdd1c admin admin] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:13:35.048751 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de910cea-3631-4071-ad2f-0348419bdd1c admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:13:35.049023 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de910cea-3631-4071-ad2f-0348419bdd1c admin admin] PATH_INFO: `/v3/projects` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:13:35.049701 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de910cea-3631-4071-ad2f-0348419bdd1c admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:13:35.051605 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de910cea-3631-4071-ad2f-0348419bdd1c admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:13:35.134809 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=89,keystone:DELETE=7,keystone:INSERT=4 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:13:35.142563 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1418/2834] 10.222.0.27 () {60 vars in 1153 bytes} [Tue Aug 30 02:13:34 2022] POST /identity/v3/projects => generated 375 bytes in 164 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 02:13:35.146657 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad6e0bd7-c84d-4e1e-8c42-53d24ab53c31 None None] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:13:35.158291 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad6e0bd7-c84d-4e1e-8c42-53d24ab53c31 None None] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:13:35.195338 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:13:35.213382 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5856de06-fc67-4e0b-9def-2dd606cbbb02 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:13:35.214009 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5856de06-fc67-4e0b-9def-2dd606cbbb02 admin admin] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:13:35.214285 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5856de06-fc67-4e0b-9def-2dd606cbbb02 admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:13:35.214550 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5856de06-fc67-4e0b-9def-2dd606cbbb02 admin admin] PATH_INFO: `/v3/users` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:13:35.215858 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5856de06-fc67-4e0b-9def-2dd606cbbb02 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:13:35.217834 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5856de06-fc67-4e0b-9def-2dd606cbbb02 admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:13:35.291847 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1417/2835] 10.222.0.27 () {60 vars in 1144 bytes} [Tue Aug 30 02:13:35 2022] POST /identity/v3/users => generated 294 bytes in 146 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 02:13:35.295928 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de910cea-3631-4071-ad2f-0348419bdd1c admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:13:35.306129 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de910cea-3631-4071-ad2f-0348419bdd1c admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:13:35.311529 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0d96afb-bbcc-4eb7-b268-9eb9a0021d64 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:13:35.312131 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0d96afb-bbcc-4eb7-b268-9eb9a0021d64 admin admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:13:35.312407 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0d96afb-bbcc-4eb7-b268-9eb9a0021d64 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:13:35.312676 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0d96afb-bbcc-4eb7-b268-9eb9a0021d64 admin admin] PATH_INFO: `/v3/roles` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:13:35.313340 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0d96afb-bbcc-4eb7-b268-9eb9a0021d64 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:13:35.314971 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0d96afb-bbcc-4eb7-b268-9eb9a0021d64 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:13:35.326372 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1419/2836] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Aug 30 02:13:35 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 02:13:35.330464 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5856de06-fc67-4e0b-9def-2dd606cbbb02 admin admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:13:35.341106 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5856de06-fc67-4e0b-9def-2dd606cbbb02 admin admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:13:35.345766 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46a95fcb-bf32-4717-9084-326a4947d3fb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:13:35.346411 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46a95fcb-bf32-4717-9084-326a4947d3fb admin admin] REQUEST_METHOD: `PUT` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:13:35.346688 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46a95fcb-bf32-4717-9084-326a4947d3fb admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:13:35.346963 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46a95fcb-bf32-4717-9084-326a4947d3fb admin admin] PATH_INFO: `/v3/projects/1d078c33edb441199db33a8099300ac3/users/9459f973ffaa4b61b105cd005268e277/roles/ead5fac86a0747ea8823a82efcea8ed7` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:13:35.362000 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46a95fcb-bf32-4717-9084-326a4947d3fb admin admin] RBAC: Authorizing `identity:create_grant(project_id=1d078c33edb441199db33a8099300ac3, user_id=9459f973ffaa4b61b105cd005268e277, role_id=ead5fac86a0747ea8823a82efcea8ed7)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:13:35.364676 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46a95fcb-bf32-4717-9084-326a4947d3fb admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:13:35.435421 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1418/2837] 10.222.0.27 () {60 vars in 1483 bytes} [Tue Aug 30 02:13:35 2022] PUT /identity/v3/projects/1d078c33edb441199db33a8099300ac3/users/9459f973ffaa4b61b105cd005268e277/roles/ead5fac86a0747ea8823a82efcea8ed7 => generated 0 bytes in 105 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 02:13:35.439547 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0d96afb-bbcc-4eb7-b268-9eb9a0021d64 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:13:35.449276 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0d96afb-bbcc-4eb7-b268-9eb9a0021d64 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:13:35.506410 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6ba7c78-66fe-4abe-bfcd-ecea05af9203 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:13:35.507005 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6ba7c78-66fe-4abe-bfcd-ecea05af9203 admin admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:13:35.507283 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6ba7c78-66fe-4abe-bfcd-ecea05af9203 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:13:35.507553 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6ba7c78-66fe-4abe-bfcd-ecea05af9203 admin admin] PATH_INFO: `/v3/roles` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:13:35.508187 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6ba7c78-66fe-4abe-bfcd-ecea05af9203 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:13:35.509820 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6ba7c78-66fe-4abe-bfcd-ecea05af9203 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:13:35.520638 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1420/2838] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Aug 30 02:13:35 2022] GET /identity/v3/roles => generated 1420 bytes in 82 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 02:13:35.524820 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46a95fcb-bf32-4717-9084-326a4947d3fb admin admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:13:35.535117 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46a95fcb-bf32-4717-9084-326a4947d3fb admin admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:13:35.539712 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebe08908-3bda-4b1b-b2b1-530e99d75279 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:13:35.540371 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebe08908-3bda-4b1b-b2b1-530e99d75279 admin admin] REQUEST_METHOD: `PUT` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:13:35.540648 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebe08908-3bda-4b1b-b2b1-530e99d75279 admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:13:35.540915 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebe08908-3bda-4b1b-b2b1-530e99d75279 admin admin] PATH_INFO: `/v3/projects/1d078c33edb441199db33a8099300ac3/users/9459f973ffaa4b61b105cd005268e277/roles/ead5fac86a0747ea8823a82efcea8ed7` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:13:35.542816 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebe08908-3bda-4b1b-b2b1-530e99d75279 admin admin] RBAC: Authorizing `identity:create_grant(project_id=1d078c33edb441199db33a8099300ac3, user_id=9459f973ffaa4b61b105cd005268e277, role_id=ead5fac86a0747ea8823a82efcea8ed7)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:13:35.545671 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebe08908-3bda-4b1b-b2b1-530e99d75279 admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:13:35.555662 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1419/2839] 10.222.0.27 () {60 vars in 1483 bytes} [Tue Aug 30 02:13:35 2022] PUT /identity/v3/projects/1d078c33edb441199db33a8099300ac3/users/9459f973ffaa4b61b105cd005268e277/roles/ead5fac86a0747ea8823a82efcea8ed7 => generated 0 bytes in 31 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 02:13:35.566540 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04d130e8-cd17-4a36-aaf0-8197aa52e172 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:13:35.566827 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04d130e8-cd17-4a36-aaf0-8197aa52e172 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:13:35.567101 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04d130e8-cd17-4a36-aaf0-8197aa52e172 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:13:35.595512 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-04d130e8-cd17-4a36-aaf0-8197aa52e172 None None] MFA Rules not processed for user `9459f973ffaa4b61b105cd005268e277`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:13:35.596736 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-04d130e8-cd17-4a36-aaf0-8197aa52e172 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:13:35.663207 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1421/2840] 10.222.0.27 () {58 vars in 958 bytes} [Tue Aug 30 02:13:35 2022] POST /identity/v3/auth/tokens => generated 3319 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 02:13:35.674619 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebe08908-3bda-4b1b-b2b1-530e99d75279 admin admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:13:35.676320 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-ebe08908-3bda-4b1b-b2b1-530e99d75279 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:13:35.736774 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebe08908-3bda-4b1b-b2b1-530e99d75279 admin admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:13:35.740445 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8c197ec-388d-4853-853e-6c78b027e0fd service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ab936b3767a8417aa80c8c8f1b84b667', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4562f3951fbe465280073e60a11e44e2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:13:35.741047 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8c197ec-388d-4853-853e-6c78b027e0fd service cinder] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:13:35.741323 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8c197ec-388d-4853-853e-6c78b027e0fd service cinder] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:13:35.741621 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8c197ec-388d-4853-853e-6c78b027e0fd service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:13:35.796468 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8c197ec-388d-4853-853e-6c78b027e0fd service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:13:35.797915 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8c197ec-388d-4853-853e-6c78b027e0fd service cinder] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:13:35.806728 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1420/2841] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Aug 30 02:13:35 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 133 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 02:13:42.099333 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04d130e8-cd17-4a36-aaf0-8197aa52e172 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:13:42.100710 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-04d130e8-cd17-4a36-aaf0-8197aa52e172 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:13:42.112888 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04d130e8-cd17-4a36-aaf0-8197aa52e172 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:13:42.116398 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70fa2131-e049-4224-ad89-02ddb8465db8 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ab936b3767a8417aa80c8c8f1b84b667', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4562f3951fbe465280073e60a11e44e2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:13:42.116990 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70fa2131-e049-4224-ad89-02ddb8465db8 service cinder] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:13:42.117268 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70fa2131-e049-4224-ad89-02ddb8465db8 service cinder] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:13:42.117569 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70fa2131-e049-4224-ad89-02ddb8465db8 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:13:42.126931 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70fa2131-e049-4224-ad89-02ddb8465db8 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:13:42.128355 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70fa2131-e049-4224-ad89-02ddb8465db8 service cinder] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:13:42.138516 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1422/2842] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Aug 30 02:13:42 2022] GET /identity/v3/auth/tokens => generated 3319 bytes in 40 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 02:13:45.802898 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=74,keystone:INSERT=5 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:13:52.133557 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=75 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:13:52.631692 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8074f1f1-862b-4f78-9c1a-fa978123c031 None None] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:13:52.632141 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8074f1f1-862b-4f78-9c1a-fa978123c031 None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:13:52.632255 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8074f1f1-862b-4f78-9c1a-fa978123c031 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:13:52.651305 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.auth.core [None req-8074f1f1-862b-4f78-9c1a-fa978123c031 None None] MFA Rules not processed for user `40e0086c1b924e7f86bd814bb45be3d9`. Rule list: `[]` (Enabled: `True`). {{(pid=81578) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:13:52.653215 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-8074f1f1-862b-4f78-9c1a-fa978123c031 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:13:52.706350 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1421/2843] 10.222.0.27 () {58 vars in 958 bytes} [Tue Aug 30 02:13:52 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 76 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 02:13:53.024578 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70fa2131-e049-4224-ad89-02ddb8465db8 service cinder] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:13:53.026123 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-70fa2131-e049-4224-ad89-02ddb8465db8 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:13:53.087999 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70fa2131-e049-4224-ad89-02ddb8465db8 service cinder] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:13:53.095136 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d27c4307-4c9c-44e4-8773-a19fd75257af service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f3ea1580c56f4814a92bbb2acf542905', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4562f3951fbe465280073e60a11e44e2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:13:53.095136 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d27c4307-4c9c-44e4-8773-a19fd75257af service neutron] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:13:53.095136 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d27c4307-4c9c-44e4-8773-a19fd75257af service neutron] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:13:53.095136 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d27c4307-4c9c-44e4-8773-a19fd75257af service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:13:53.095980 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-d27c4307-4c9c-44e4-8773-a19fd75257af service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:13:53.104102 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d27c4307-4c9c-44e4-8773-a19fd75257af service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:13:53.106114 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d27c4307-4c9c-44e4-8773-a19fd75257af service neutron] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:13:53.114933 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1423/2844] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Aug 30 02:13:53 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 91 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 02:13:54.892392 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8074f1f1-862b-4f78-9c1a-fa978123c031 None None] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:13:54.894183 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-8074f1f1-862b-4f78-9c1a-fa978123c031 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:13:54.966087 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8074f1f1-862b-4f78-9c1a-fa978123c031 None None] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:13:54.970968 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d6e9bde-af80-4576-9db1-9c273764c002 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '379536ff06fd4f719899e5dde732e63d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4562f3951fbe465280073e60a11e44e2', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:13:54.971581 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d6e9bde-af80-4576-9db1-9c273764c002 service placement] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:13:54.971861 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d6e9bde-af80-4576-9db1-9c273764c002 service placement] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:13:54.972131 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d6e9bde-af80-4576-9db1-9c273764c002 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:13:54.973873 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-0d6e9bde-af80-4576-9db1-9c273764c002 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:13:54.985954 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d6e9bde-af80-4576-9db1-9c273764c002 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:13:54.987511 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d6e9bde-af80-4576-9db1-9c273764c002 service placement] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:13:54.998059 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1422/2845] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Aug 30 02:13:54 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 106 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 02:14:03.110924 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=26 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:14:04.993062 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=57 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:14:09.315598 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d27c4307-4c9c-44e4-8773-a19fd75257af service neutron] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:14:09.377331 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d27c4307-4c9c-44e4-8773-a19fd75257af service neutron] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:14:09.381873 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a355b73-df9a-4b30-8f7e-e2457e992844 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:14:09.382497 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a355b73-df9a-4b30-8f7e-e2457e992844 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:14:09.382775 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a355b73-df9a-4b30-8f7e-e2457e992844 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:14:09.383046 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a355b73-df9a-4b30-8f7e-e2457e992844 admin admin] PATH_INFO: `/v3/users/c47327e1276f447f83a96368d2d6e651` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:14:09.384201 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a355b73-df9a-4b30-8f7e-e2457e992844 admin admin] RBAC: Authorizing `identity:delete_user(user_id=c47327e1276f447f83a96368d2d6e651)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:14:09.387083 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a355b73-df9a-4b30-8f7e-e2457e992844 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:14:09.548334 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-2a355b73-df9a-4b30-8f7e-e2457e992844 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'c47327e1276f447f83a96368d2d6e651'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:14:09.548973 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-2a355b73-df9a-4b30-8f7e-e2457e992844 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'c47327e1276f447f83a96368d2d6e651'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:14:09.549656 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-2a355b73-df9a-4b30-8f7e-e2457e992844 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'c47327e1276f447f83a96368d2d6e651'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:14:09.562708 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-2a355b73-df9a-4b30-8f7e-e2457e992844 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'c47327e1276f447f83a96368d2d6e651'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:14:09.571196 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-2a355b73-df9a-4b30-8f7e-e2457e992844 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'c47327e1276f447f83a96368d2d6e651'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:14:09.630625 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1424/2846] 10.222.0.27 () {58 vars in 1224 bytes} [Tue Aug 30 02:14:09 2022] DELETE /identity/v3/users/c47327e1276f447f83a96368d2d6e651 => generated 0 bytes in 316 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 02:14:09.635083 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad790215-5707-48b5-b3b2-e1e0775fb78f None None] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:14:09.635566 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad790215-5707-48b5-b3b2-e1e0775fb78f None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:14:09.635678 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad790215-5707-48b5-b3b2-e1e0775fb78f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:14:09.654639 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.auth.core [None req-ad790215-5707-48b5-b3b2-e1e0775fb78f None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81578) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:14:09.655840 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-ad790215-5707-48b5-b3b2-e1e0775fb78f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:14:09.717331 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1423/2847] 10.222.0.27 () {58 vars in 958 bytes} [Tue Aug 30 02:14:09 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 02:14:09.727022 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a355b73-df9a-4b30-8f7e-e2457e992844 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:14:09.728424 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-2a355b73-df9a-4b30-8f7e-e2457e992844 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:14:09.789806 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a355b73-df9a-4b30-8f7e-e2457e992844 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:14:09.793566 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74be4183-51ed-4d9d-8ec7-f17ac9ccc0b1 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f3ea1580c56f4814a92bbb2acf542905', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4562f3951fbe465280073e60a11e44e2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:14:09.794168 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74be4183-51ed-4d9d-8ec7-f17ac9ccc0b1 service neutron] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:14:09.794448 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74be4183-51ed-4d9d-8ec7-f17ac9ccc0b1 service neutron] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:14:09.794719 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74be4183-51ed-4d9d-8ec7-f17ac9ccc0b1 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:14:09.803761 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74be4183-51ed-4d9d-8ec7-f17ac9ccc0b1 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:14:09.805127 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74be4183-51ed-4d9d-8ec7-f17ac9ccc0b1 service neutron] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:14:09.815080 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1425/2848] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Aug 30 02:14:09 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 89 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 02:14:10.204302 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad790215-5707-48b5-b3b2-e1e0775fb78f None None] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:14:10.205878 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-ad790215-5707-48b5-b3b2-e1e0775fb78f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:14:10.218172 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad790215-5707-48b5-b3b2-e1e0775fb78f None None] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:14:10.222945 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32b8fab6-b6d2-4ffa-bcd1-38807bb79500 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:14:10.223575 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32b8fab6-b6d2-4ffa-bcd1-38807bb79500 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:14:10.223854 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32b8fab6-b6d2-4ffa-bcd1-38807bb79500 admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:14:10.224151 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32b8fab6-b6d2-4ffa-bcd1-38807bb79500 admin admin] PATH_INFO: `/v3/projects/885776e9f06e4b798f7fcd1c6e12339f` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:14:10.225422 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32b8fab6-b6d2-4ffa-bcd1-38807bb79500 admin admin] RBAC: Authorizing `identity:delete_project(project_id=885776e9f06e4b798f7fcd1c6e12339f)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:14:10.227339 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32b8fab6-b6d2-4ffa-bcd1-38807bb79500 admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:14:10.340659 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-32b8fab6-b6d2-4ffa-bcd1-38807bb79500 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '885776e9f06e4b798f7fcd1c6e12339f', 'request_id': 'req-32b8fab6-b6d2-4ffa-bcd1-38807bb79500'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:14:10.346556 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-32b8fab6-b6d2-4ffa-bcd1-38807bb79500 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '885776e9f06e4b798f7fcd1c6e12339f', 'request_id': 'req-32b8fab6-b6d2-4ffa-bcd1-38807bb79500'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:14:10.353192 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-32b8fab6-b6d2-4ffa-bcd1-38807bb79500 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '885776e9f06e4b798f7fcd1c6e12339f', 'request_id': 'req-32b8fab6-b6d2-4ffa-bcd1-38807bb79500'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:14:10.425811 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-32b8fab6-b6d2-4ffa-bcd1-38807bb79500 admin admin] The token cache is being invalidate because project 885776e9f06e4b798f7fcd1c6e12339f was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81578) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 30 02:14:10.426101 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-32b8fab6-b6d2-4ffa-bcd1-38807bb79500 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:14:10.426754 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-32b8fab6-b6d2-4ffa-bcd1-38807bb79500 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:14:10.428215 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1424/2849] 10.222.0.27 () {58 vars in 1233 bytes} [Tue Aug 30 02:14:10 2022] DELETE /identity/v3/projects/885776e9f06e4b798f7fcd1c6e12339f => generated 0 bytes in 224 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 02:14:10.432248 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74be4183-51ed-4d9d-8ec7-f17ac9ccc0b1 service neutron] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:14:10.442614 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74be4183-51ed-4d9d-8ec7-f17ac9ccc0b1 service neutron] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:14:10.460904 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:14:10.501055 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1562c13-a221-41e3-9e54-70b95248ec1f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:14:10.506189 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1562c13-a221-41e3-9e54-70b95248ec1f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:14:10.506189 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1562c13-a221-41e3-9e54-70b95248ec1f admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:14:10.506189 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1562c13-a221-41e3-9e54-70b95248ec1f admin admin] PATH_INFO: `/v3/users/9459f973ffaa4b61b105cd005268e277` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:14:10.506189 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1562c13-a221-41e3-9e54-70b95248ec1f admin admin] RBAC: Authorizing `identity:delete_user(user_id=9459f973ffaa4b61b105cd005268e277)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:14:10.506189 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1562c13-a221-41e3-9e54-70b95248ec1f admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:14:10.677739 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-b1562c13-a221-41e3-9e54-70b95248ec1f admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '9459f973ffaa4b61b105cd005268e277'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:14:10.678315 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-b1562c13-a221-41e3-9e54-70b95248ec1f admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '9459f973ffaa4b61b105cd005268e277'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:14:10.678916 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-b1562c13-a221-41e3-9e54-70b95248ec1f admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '9459f973ffaa4b61b105cd005268e277'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:14:10.691258 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-b1562c13-a221-41e3-9e54-70b95248ec1f admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '9459f973ffaa4b61b105cd005268e277'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:14:10.698006 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-b1562c13-a221-41e3-9e54-70b95248ec1f admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '9459f973ffaa4b61b105cd005268e277'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:14:10.791899 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1426/2850] 10.222.0.27 () {58 vars in 1224 bytes} [Tue Aug 30 02:14:10 2022] DELETE /identity/v3/users/9459f973ffaa4b61b105cd005268e277 => generated 0 bytes in 360 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 02:14:11.190979 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32b8fab6-b6d2-4ffa-bcd1-38807bb79500 admin admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:14:11.193191 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-32b8fab6-b6d2-4ffa-bcd1-38807bb79500 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:14:11.206939 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32b8fab6-b6d2-4ffa-bcd1-38807bb79500 admin admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:14:11.274002 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-adccedd3-b617-4550-8c1d-b1b526d1c017 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:14:11.274639 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adccedd3-b617-4550-8c1d-b1b526d1c017 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:14:11.274920 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adccedd3-b617-4550-8c1d-b1b526d1c017 admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:14:11.275192 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adccedd3-b617-4550-8c1d-b1b526d1c017 admin admin] PATH_INFO: `/v3/projects/1d078c33edb441199db33a8099300ac3` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:14:11.276330 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-adccedd3-b617-4550-8c1d-b1b526d1c017 admin admin] RBAC: Authorizing `identity:delete_project(project_id=1d078c33edb441199db33a8099300ac3)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:14:11.278403 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-adccedd3-b617-4550-8c1d-b1b526d1c017 admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:14:11.375268 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=91,keystone:DELETE=8,keystone:INSERT=1 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:14:11.391421 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-adccedd3-b617-4550-8c1d-b1b526d1c017 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '1d078c33edb441199db33a8099300ac3', 'request_id': 'req-adccedd3-b617-4550-8c1d-b1b526d1c017'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:14:11.396739 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-adccedd3-b617-4550-8c1d-b1b526d1c017 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '1d078c33edb441199db33a8099300ac3', 'request_id': 'req-adccedd3-b617-4550-8c1d-b1b526d1c017'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:14:11.402891 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-adccedd3-b617-4550-8c1d-b1b526d1c017 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '1d078c33edb441199db33a8099300ac3', 'request_id': 'req-adccedd3-b617-4550-8c1d-b1b526d1c017'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:14:11.485433 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-adccedd3-b617-4550-8c1d-b1b526d1c017 admin admin] The token cache is being invalidate because project 1d078c33edb441199db33a8099300ac3 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81578) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 30 02:14:11.485680 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-adccedd3-b617-4550-8c1d-b1b526d1c017 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:14:11.486310 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-adccedd3-b617-4550-8c1d-b1b526d1c017 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:14:11.487797 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1425/2851] 10.222.0.27 () {58 vars in 1233 bytes} [Tue Aug 30 02:14:11 2022] DELETE /identity/v3/projects/1d078c33edb441199db33a8099300ac3 => generated 0 bytes in 298 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 02:14:11.492657 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-441dfe1a-2ff6-46f3-9777-2f971051644e None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:14:11.493145 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-441dfe1a-2ff6-46f3-9777-2f971051644e None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:14:11.493261 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-441dfe1a-2ff6-46f3-9777-2f971051644e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:14:11.513856 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-441dfe1a-2ff6-46f3-9777-2f971051644e None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:14:11.519448 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-441dfe1a-2ff6-46f3-9777-2f971051644e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:14:11.574420 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1427/2852] 10.222.0.27 () {58 vars in 958 bytes} [Tue Aug 30 02:14:11 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 02:14:11.638933 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dba03b5-7a12-4420-9944-87ca02823f94 None None] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:14:11.639207 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dba03b5-7a12-4420-9944-87ca02823f94 None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:14:11.639482 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dba03b5-7a12-4420-9944-87ca02823f94 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:14:11.659326 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.auth.core [None req-3dba03b5-7a12-4420-9944-87ca02823f94 None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81578) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:14:11.660529 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-3dba03b5-7a12-4420-9944-87ca02823f94 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:14:11.669004 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1426/2853] 10.222.0.27 () {58 vars in 958 bytes} [Tue Aug 30 02:14:11 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 02:14:11.677081 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-441dfe1a-2ff6-46f3-9777-2f971051644e None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:14:11.688001 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-441dfe1a-2ff6-46f3-9777-2f971051644e None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:14:11.692363 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8783627-86eb-4941-a84a-9da945f49c9c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:14:11.692970 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8783627-86eb-4941-a84a-9da945f49c9c admin admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:14:11.693243 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8783627-86eb-4941-a84a-9da945f49c9c admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:14:11.693537 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8783627-86eb-4941-a84a-9da945f49c9c admin admin] PATH_INFO: `/v3/domains` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:14:11.694208 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8783627-86eb-4941-a84a-9da945f49c9c admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 02:14:11.694507 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8783627-86eb-4941-a84a-9da945f49c9c admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:14:11.696150 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8783627-86eb-4941-a84a-9da945f49c9c admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:14:11.708375 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1428/2854] 10.222.0.27 () {58 vars in 1153 bytes} [Tue Aug 30 02:14:11 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 02:14:11.712745 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3dba03b5-7a12-4420-9944-87ca02823f94 None None] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:14:11.722918 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3dba03b5-7a12-4420-9944-87ca02823f94 None None] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:14:11.727643 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4aa0fad2-902b-4a45-a86b-966368e82a40 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:14:11.737094 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4aa0fad2-902b-4a45-a86b-966368e82a40 admin admin] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:14:11.737094 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4aa0fad2-902b-4a45-a86b-966368e82a40 admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:14:11.737094 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4aa0fad2-902b-4a45-a86b-966368e82a40 admin admin] PATH_INFO: `/v3/projects` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:14:11.737094 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4aa0fad2-902b-4a45-a86b-966368e82a40 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:14:11.737094 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4aa0fad2-902b-4a45-a86b-966368e82a40 admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:14:11.810194 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1427/2855] 10.222.0.27 () {60 vars in 1153 bytes} [Tue Aug 30 02:14:11 2022] POST /identity/v3/projects => generated 375 bytes in 98 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 30 02:14:11.814395 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8783627-86eb-4941-a84a-9da945f49c9c admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:14:11.824630 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8783627-86eb-4941-a84a-9da945f49c9c admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:14:11.849359 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:14:11.880002 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c2f5404-e4f1-49a2-bb08-f37bcda792ec admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:14:11.880599 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c2f5404-e4f1-49a2-bb08-f37bcda792ec admin admin] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:14:11.880875 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c2f5404-e4f1-49a2-bb08-f37bcda792ec admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:14:11.881172 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c2f5404-e4f1-49a2-bb08-f37bcda792ec admin admin] PATH_INFO: `/v3/users` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:14:11.881825 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c2f5404-e4f1-49a2-bb08-f37bcda792ec admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:14:11.883729 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c2f5404-e4f1-49a2-bb08-f37bcda792ec admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:14:11.977232 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1429/2856] 10.222.0.27 () {60 vars in 1144 bytes} [Tue Aug 30 02:14:11 2022] POST /identity/v3/users => generated 288 bytes in 163 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 02:14:11.981430 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4aa0fad2-902b-4a45-a86b-966368e82a40 admin admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:14:11.992423 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4aa0fad2-902b-4a45-a86b-966368e82a40 admin admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:14:11.997735 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4369a54f-bb92-4b2b-b228-2acc5e0b29a6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:14:11.998327 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4369a54f-bb92-4b2b-b228-2acc5e0b29a6 admin admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:14:11.998603 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4369a54f-bb92-4b2b-b228-2acc5e0b29a6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:14:11.998874 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4369a54f-bb92-4b2b-b228-2acc5e0b29a6 admin admin] PATH_INFO: `/v3/roles` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:14:11.999559 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4369a54f-bb92-4b2b-b228-2acc5e0b29a6 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:14:12.001137 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4369a54f-bb92-4b2b-b228-2acc5e0b29a6 admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:14:12.012145 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1428/2857] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Aug 30 02:14:11 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 02:14:12.016383 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c2f5404-e4f1-49a2-bb08-f37bcda792ec admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:14:12.026087 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c2f5404-e4f1-49a2-bb08-f37bcda792ec admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:14:12.031455 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36dfe326-0a84-4ed8-96d9-70e4500ae8bb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:14:12.032043 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36dfe326-0a84-4ed8-96d9-70e4500ae8bb admin admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:14:12.032318 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36dfe326-0a84-4ed8-96d9-70e4500ae8bb admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:14:12.032713 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36dfe326-0a84-4ed8-96d9-70e4500ae8bb admin admin] PATH_INFO: `/v3/roles` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:14:12.033354 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36dfe326-0a84-4ed8-96d9-70e4500ae8bb admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:14:12.034973 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36dfe326-0a84-4ed8-96d9-70e4500ae8bb admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:14:12.045644 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1430/2858] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Aug 30 02:14:12 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 02:14:12.049899 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4369a54f-bb92-4b2b-b228-2acc5e0b29a6 admin admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:14:12.060497 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4369a54f-bb92-4b2b-b228-2acc5e0b29a6 admin admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:14:12.065133 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb2f0884-9fc6-436e-ac9e-3bb5bebc13dd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:14:12.065806 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb2f0884-9fc6-436e-ac9e-3bb5bebc13dd admin admin] REQUEST_METHOD: `PUT` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:14:12.066085 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb2f0884-9fc6-436e-ac9e-3bb5bebc13dd admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:14:12.066357 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb2f0884-9fc6-436e-ac9e-3bb5bebc13dd admin admin] PATH_INFO: `/v3/projects/01cdd6ac42a74591914f216cf68852a9/users/07dad0417900479fa42b46aef306a280/roles/1fb81afcb9f245f3b989add94598a063` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:14:12.081299 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb2f0884-9fc6-436e-ac9e-3bb5bebc13dd admin admin] RBAC: Authorizing `identity:create_grant(project_id=01cdd6ac42a74591914f216cf68852a9, user_id=07dad0417900479fa42b46aef306a280, role_id=1fb81afcb9f245f3b989add94598a063)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:14:12.083967 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb2f0884-9fc6-436e-ac9e-3bb5bebc13dd admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:14:12.157088 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1429/2859] 10.222.0.27 () {60 vars in 1483 bytes} [Tue Aug 30 02:14:12 2022] PUT /identity/v3/projects/01cdd6ac42a74591914f216cf68852a9/users/07dad0417900479fa42b46aef306a280/roles/1fb81afcb9f245f3b989add94598a063 => generated 0 bytes in 108 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 02:14:12.166995 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dbc05b0-6b74-4058-9e7a-124ea89689d4 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:14:12.167253 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dbc05b0-6b74-4058-9e7a-124ea89689d4 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:14:12.167528 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dbc05b0-6b74-4058-9e7a-124ea89689d4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:14:12.196628 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-9dbc05b0-6b74-4058-9e7a-124ea89689d4 None None] MFA Rules not processed for user `07dad0417900479fa42b46aef306a280`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:14:12.197877 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-9dbc05b0-6b74-4058-9e7a-124ea89689d4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:14:12.270179 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1431/2860] 10.222.0.27 () {58 vars in 958 bytes} [Tue Aug 30 02:14:12 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 02:14:12.279593 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb2f0884-9fc6-436e-ac9e-3bb5bebc13dd admin admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:14:12.281329 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-fb2f0884-9fc6-436e-ac9e-3bb5bebc13dd admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:14:12.365828 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb2f0884-9fc6-436e-ac9e-3bb5bebc13dd admin admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:14:12.372435 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30363195-c8c3-4254-ad98-48e526d1fc8a service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3caff62093d748f48596574836e12e42', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4562f3951fbe465280073e60a11e44e2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:14:12.373060 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30363195-c8c3-4254-ad98-48e526d1fc8a service glance] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:14:12.373328 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30363195-c8c3-4254-ad98-48e526d1fc8a service glance] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:14:12.373631 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30363195-c8c3-4254-ad98-48e526d1fc8a service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:14:12.382649 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30363195-c8c3-4254-ad98-48e526d1fc8a service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:14:12.384119 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30363195-c8c3-4254-ad98-48e526d1fc8a service glance] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:14:12.393711 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1430/2861] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Aug 30 02:14:12 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 114 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 02:14:12.490481 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9dbc05b0-6b74-4058-9e7a-124ea89689d4 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:14:12.491915 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-9dbc05b0-6b74-4058-9e7a-124ea89689d4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:14:12.574658 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9dbc05b0-6b74-4058-9e7a-124ea89689d4 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:14:12.587750 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21d8ba63-fce2-4e4e-905c-ab1f7a3da22a service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd64f2e71361644589bec8a06e6eceed1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4562f3951fbe465280073e60a11e44e2', 'project_domain_id': 'default', 'roles': ['member', 'service', 'admin', 'ResellerAdmin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:14:12.588671 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21d8ba63-fce2-4e4e-905c-ab1f7a3da22a service nova] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:14:12.589051 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21d8ba63-fce2-4e4e-905c-ab1f7a3da22a service nova] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:14:12.589421 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21d8ba63-fce2-4e4e-905c-ab1f7a3da22a service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:14:12.591819 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-21d8ba63-fce2-4e4e-905c-ab1f7a3da22a service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:14:12.606645 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21d8ba63-fce2-4e4e-905c-ab1f7a3da22a service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:14:12.608230 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21d8ba63-fce2-4e4e-905c-ab1f7a3da22a service nova] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:14:12.620881 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1432/2862] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Aug 30 02:14:12 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 131 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 02:14:22.389318 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:DELETE=6,keystone:SELECT=76,keystone:INSERT=3 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:14:22.615091 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=87,keystone:INSERT=3 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:14:28.488656 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30363195-c8c3-4254-ad98-48e526d1fc8a service glance] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:14:28.550211 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30363195-c8c3-4254-ad98-48e526d1fc8a service glance] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:14:28.554614 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0be2c12-08fe-4644-a060-bd02f82f9d36 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:14:28.555230 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0be2c12-08fe-4644-a060-bd02f82f9d36 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:14:28.555504 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0be2c12-08fe-4644-a060-bd02f82f9d36 admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:14:28.555773 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0be2c12-08fe-4644-a060-bd02f82f9d36 admin admin] PATH_INFO: `/v3/users/07dad0417900479fa42b46aef306a280` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:14:28.556868 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0be2c12-08fe-4644-a060-bd02f82f9d36 admin admin] RBAC: Authorizing `identity:delete_user(user_id=07dad0417900479fa42b46aef306a280)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:14:28.558840 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0be2c12-08fe-4644-a060-bd02f82f9d36 admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:14:28.758650 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-c0be2c12-08fe-4644-a060-bd02f82f9d36 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '07dad0417900479fa42b46aef306a280'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:14:28.759375 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-c0be2c12-08fe-4644-a060-bd02f82f9d36 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '07dad0417900479fa42b46aef306a280'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:14:28.842397 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-c0be2c12-08fe-4644-a060-bd02f82f9d36 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '07dad0417900479fa42b46aef306a280'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:14:28.843021 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-c0be2c12-08fe-4644-a060-bd02f82f9d36 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '07dad0417900479fa42b46aef306a280'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:14:28.850163 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-c0be2c12-08fe-4644-a060-bd02f82f9d36 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '07dad0417900479fa42b46aef306a280'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:14:28.867515 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1431/2863] 10.222.0.27 () {58 vars in 1224 bytes} [Tue Aug 30 02:14:28 2022] DELETE /identity/v3/users/07dad0417900479fa42b46aef306a280 => generated 0 bytes in 379 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 02:14:28.872032 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab44cf26-c4f6-4912-b732-aa16cd1718a8 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:14:28.872523 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab44cf26-c4f6-4912-b732-aa16cd1718a8 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:14:28.872657 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab44cf26-c4f6-4912-b732-aa16cd1718a8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:14:28.892425 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-ab44cf26-c4f6-4912-b732-aa16cd1718a8 None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:14:28.893653 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-ab44cf26-c4f6-4912-b732-aa16cd1718a8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:14:28.952649 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1433/2864] 10.222.0.27 () {58 vars in 958 bytes} [Tue Aug 30 02:14:28 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 82 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 02:14:28.962742 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0be2c12-08fe-4644-a060-bd02f82f9d36 admin admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:14:28.964987 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-c0be2c12-08fe-4644-a060-bd02f82f9d36 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:14:29.022300 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0be2c12-08fe-4644-a060-bd02f82f9d36 admin admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:14:29.026475 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0eb67527-8577-48c5-b246-d85611e3ee2f service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f3ea1580c56f4814a92bbb2acf542905', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4562f3951fbe465280073e60a11e44e2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:14:29.027070 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0eb67527-8577-48c5-b246-d85611e3ee2f service neutron] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:14:29.027353 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0eb67527-8577-48c5-b246-d85611e3ee2f service neutron] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:14:29.027625 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0eb67527-8577-48c5-b246-d85611e3ee2f service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:14:29.036512 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0eb67527-8577-48c5-b246-d85611e3ee2f service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:14:29.037976 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0eb67527-8577-48c5-b246-d85611e3ee2f service neutron] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:14:29.047404 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1432/2865] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Aug 30 02:14:28 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 85 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 02:14:29.501431 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab44cf26-c4f6-4912-b732-aa16cd1718a8 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:14:29.503794 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-ab44cf26-c4f6-4912-b732-aa16cd1718a8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:14:29.516293 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab44cf26-c4f6-4912-b732-aa16cd1718a8 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:14:29.520960 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f1abd4e-e516-465b-ac4b-ac61d7a65de0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:14:29.521664 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f1abd4e-e516-465b-ac4b-ac61d7a65de0 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:14:29.521944 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f1abd4e-e516-465b-ac4b-ac61d7a65de0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:14:29.522216 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f1abd4e-e516-465b-ac4b-ac61d7a65de0 admin admin] PATH_INFO: `/v3/projects/01cdd6ac42a74591914f216cf68852a9` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:14:29.523325 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f1abd4e-e516-465b-ac4b-ac61d7a65de0 admin admin] RBAC: Authorizing `identity:delete_project(project_id=01cdd6ac42a74591914f216cf68852a9)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:14:29.525231 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f1abd4e-e516-465b-ac4b-ac61d7a65de0 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:14:29.648565 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-0f1abd4e-e516-465b-ac4b-ac61d7a65de0 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '01cdd6ac42a74591914f216cf68852a9', 'request_id': 'req-0f1abd4e-e516-465b-ac4b-ac61d7a65de0'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:14:29.654604 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-0f1abd4e-e516-465b-ac4b-ac61d7a65de0 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '01cdd6ac42a74591914f216cf68852a9', 'request_id': 'req-0f1abd4e-e516-465b-ac4b-ac61d7a65de0'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:14:29.660778 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-0f1abd4e-e516-465b-ac4b-ac61d7a65de0 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '01cdd6ac42a74591914f216cf68852a9', 'request_id': 'req-0f1abd4e-e516-465b-ac4b-ac61d7a65de0'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:14:29.751487 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-0f1abd4e-e516-465b-ac4b-ac61d7a65de0 admin admin] The token cache is being invalidate because project 01cdd6ac42a74591914f216cf68852a9 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81577) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 30 02:14:29.751776 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-0f1abd4e-e516-465b-ac4b-ac61d7a65de0 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:14:29.752398 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-0f1abd4e-e516-465b-ac4b-ac61d7a65de0 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:14:29.753987 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1434/2866] 10.222.0.27 () {58 vars in 1233 bytes} [Tue Aug 30 02:14:29 2022] DELETE /identity/v3/projects/01cdd6ac42a74591914f216cf68852a9 => generated 0 bytes in 253 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 02:14:29.758789 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc6186e6-2626-4d1e-8bbc-d2da537e04b6 None None] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:14:29.760432 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc6186e6-2626-4d1e-8bbc-d2da537e04b6 None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:14:29.760432 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc6186e6-2626-4d1e-8bbc-d2da537e04b6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:14:29.778558 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.auth.core [None req-bc6186e6-2626-4d1e-8bbc-d2da537e04b6 None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81578) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:14:29.779747 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-bc6186e6-2626-4d1e-8bbc-d2da537e04b6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:14:29.796956 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:14:29.840408 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1433/2867] 10.222.0.27 () {58 vars in 958 bytes} [Tue Aug 30 02:14:29 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 02:14:29.849507 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7719f6d-cc9d-4b44-8bfb-ac6ab1249c6b None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:14:29.849798 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7719f6d-cc9d-4b44-8bfb-ac6ab1249c6b None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:14:29.850068 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7719f6d-cc9d-4b44-8bfb-ac6ab1249c6b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:14:29.870907 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-d7719f6d-cc9d-4b44-8bfb-ac6ab1249c6b None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:14:29.872099 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-d7719f6d-cc9d-4b44-8bfb-ac6ab1249c6b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:14:29.879887 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1435/2868] 10.222.0.27 () {58 vars in 958 bytes} [Tue Aug 30 02:14:29 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 02:14:29.884145 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc6186e6-2626-4d1e-8bbc-d2da537e04b6 None None] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:14:29.894383 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc6186e6-2626-4d1e-8bbc-d2da537e04b6 None None] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:14:29.899016 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63c30d2d-bc06-4a76-a6b1-caef95bacd64 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:14:29.899561 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63c30d2d-bc06-4a76-a6b1-caef95bacd64 admin admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:14:29.899844 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63c30d2d-bc06-4a76-a6b1-caef95bacd64 admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:14:29.900145 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63c30d2d-bc06-4a76-a6b1-caef95bacd64 admin admin] PATH_INFO: `/v3/domains` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:14:29.900810 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63c30d2d-bc06-4a76-a6b1-caef95bacd64 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81578) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 02:14:29.901312 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63c30d2d-bc06-4a76-a6b1-caef95bacd64 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:14:29.903156 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63c30d2d-bc06-4a76-a6b1-caef95bacd64 admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:14:29.914755 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1434/2869] 10.222.0.27 () {58 vars in 1153 bytes} [Tue Aug 30 02:14:29 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 30 02:14:29.919851 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7719f6d-cc9d-4b44-8bfb-ac6ab1249c6b None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:14:29.928966 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7719f6d-cc9d-4b44-8bfb-ac6ab1249c6b None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:14:29.934465 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a165ead-b850-4f1d-9e6e-0994d5f69ea5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:14:29.935073 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a165ead-b850-4f1d-9e6e-0994d5f69ea5 admin admin] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:14:29.935352 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a165ead-b850-4f1d-9e6e-0994d5f69ea5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:14:29.935631 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a165ead-b850-4f1d-9e6e-0994d5f69ea5 admin admin] PATH_INFO: `/v3/projects` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:14:29.936286 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a165ead-b850-4f1d-9e6e-0994d5f69ea5 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:14:29.938213 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a165ead-b850-4f1d-9e6e-0994d5f69ea5 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:14:30.033128 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1436/2870] 10.222.0.27 () {60 vars in 1153 bytes} [Tue Aug 30 02:14:29 2022] POST /identity/v3/projects => generated 361 bytes in 114 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 02:14:30.037277 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63c30d2d-bc06-4a76-a6b1-caef95bacd64 admin admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:14:30.047926 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63c30d2d-bc06-4a76-a6b1-caef95bacd64 admin admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:14:30.105558 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d69265ec-e1cf-4b77-a961-840663183f86 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:14:30.106164 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d69265ec-e1cf-4b77-a961-840663183f86 admin admin] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:14:30.106475 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d69265ec-e1cf-4b77-a961-840663183f86 admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:14:30.106748 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d69265ec-e1cf-4b77-a961-840663183f86 admin admin] PATH_INFO: `/v3/users` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:14:30.107382 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d69265ec-e1cf-4b77-a961-840663183f86 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:14:30.109586 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d69265ec-e1cf-4b77-a961-840663183f86 admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:14:30.174393 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1435/2871] 10.222.0.27 () {60 vars in 1144 bytes} [Tue Aug 30 02:14:30 2022] POST /identity/v3/users => generated 281 bytes in 138 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 02:14:30.178369 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a165ead-b850-4f1d-9e6e-0994d5f69ea5 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:14:30.188660 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a165ead-b850-4f1d-9e6e-0994d5f69ea5 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:14:30.193382 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fae85f76-27a3-49f3-ac84-23067b0720fa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:14:30.194004 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fae85f76-27a3-49f3-ac84-23067b0720fa admin admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:14:30.194287 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fae85f76-27a3-49f3-ac84-23067b0720fa admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:14:30.194559 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fae85f76-27a3-49f3-ac84-23067b0720fa admin admin] PATH_INFO: `/v3/roles` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:14:30.195260 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fae85f76-27a3-49f3-ac84-23067b0720fa admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:14:30.196790 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fae85f76-27a3-49f3-ac84-23067b0720fa admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:14:30.206753 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1437/2872] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Aug 30 02:14:30 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 02:14:30.210654 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d69265ec-e1cf-4b77-a961-840663183f86 admin admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:14:30.221565 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d69265ec-e1cf-4b77-a961-840663183f86 admin admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:14:30.226180 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97f1fe32-fe40-4ebb-ab7c-48664697afee admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:14:30.226763 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97f1fe32-fe40-4ebb-ab7c-48664697afee admin admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:14:30.227040 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97f1fe32-fe40-4ebb-ab7c-48664697afee admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:14:30.227307 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97f1fe32-fe40-4ebb-ab7c-48664697afee admin admin] PATH_INFO: `/v3/roles` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:14:30.227940 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97f1fe32-fe40-4ebb-ab7c-48664697afee admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:14:30.230365 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97f1fe32-fe40-4ebb-ab7c-48664697afee admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:14:30.241516 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1436/2873] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Aug 30 02:14:30 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 02:14:30.245651 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fae85f76-27a3-49f3-ac84-23067b0720fa admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:14:30.255541 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fae85f76-27a3-49f3-ac84-23067b0720fa admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:14:30.260163 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bea9f502-174e-4774-b362-89c8139367c8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:14:30.260888 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bea9f502-174e-4774-b362-89c8139367c8 admin admin] REQUEST_METHOD: `PUT` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:14:30.261177 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bea9f502-174e-4774-b362-89c8139367c8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:14:30.261446 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bea9f502-174e-4774-b362-89c8139367c8 admin admin] PATH_INFO: `/v3/projects/fc9febd94ffd4099b38f052fbac58372/users/30321b42749447ea81c84d7d3c8f2094/roles/1fb81afcb9f245f3b989add94598a063` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:14:30.275706 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bea9f502-174e-4774-b362-89c8139367c8 admin admin] RBAC: Authorizing `identity:create_grant(project_id=fc9febd94ffd4099b38f052fbac58372, user_id=30321b42749447ea81c84d7d3c8f2094, role_id=1fb81afcb9f245f3b989add94598a063)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:14:30.278870 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bea9f502-174e-4774-b362-89c8139367c8 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:14:30.337632 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1438/2874] 10.222.0.27 () {60 vars in 1483 bytes} [Tue Aug 30 02:14:30 2022] PUT /identity/v3/projects/fc9febd94ffd4099b38f052fbac58372/users/30321b42749447ea81c84d7d3c8f2094/roles/1fb81afcb9f245f3b989add94598a063 => generated 0 bytes in 92 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 02:14:30.348757 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f263a97-96a0-4fbe-a8d4-7ca271d52ecc None None] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:14:30.349047 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f263a97-96a0-4fbe-a8d4-7ca271d52ecc None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:14:30.349318 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f263a97-96a0-4fbe-a8d4-7ca271d52ecc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:14:30.377307 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.auth.core [None req-9f263a97-96a0-4fbe-a8d4-7ca271d52ecc None None] MFA Rules not processed for user `30321b42749447ea81c84d7d3c8f2094`. Rule list: `[]` (Enabled: `True`). {{(pid=81578) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:14:30.378607 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-9f263a97-96a0-4fbe-a8d4-7ca271d52ecc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:14:30.443050 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1437/2875] 10.222.0.27 () {58 vars in 958 bytes} [Tue Aug 30 02:14:30 2022] POST /identity/v3/auth/tokens => generated 3238 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 02:14:30.476955 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bea9f502-174e-4774-b362-89c8139367c8 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:14:30.478730 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-bea9f502-174e-4774-b362-89c8139367c8 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:14:30.502252 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:14:30.542045 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bea9f502-174e-4774-b362-89c8139367c8 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:14:30.546009 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4100f43-8108-4d56-a722-c07e6b6f0e94 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ab936b3767a8417aa80c8c8f1b84b667', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4562f3951fbe465280073e60a11e44e2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:14:30.546623 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4100f43-8108-4d56-a722-c07e6b6f0e94 service cinder] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:14:30.546901 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4100f43-8108-4d56-a722-c07e6b6f0e94 service cinder] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:14:30.547173 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4100f43-8108-4d56-a722-c07e6b6f0e94 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:14:30.555622 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4100f43-8108-4d56-a722-c07e6b6f0e94 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:14:30.557725 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4100f43-8108-4d56-a722-c07e6b6f0e94 service cinder] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:14:30.566710 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1439/2876] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Aug 30 02:14:30 2022] GET /identity/v3/auth/tokens => generated 3238 bytes in 90 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 02:14:30.581873 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f263a97-96a0-4fbe-a8d4-7ca271d52ecc None None] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:14:30.600817 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:14:30.645875 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f263a97-96a0-4fbe-a8d4-7ca271d52ecc None None] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:14:30.651465 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4400b329-0424-4d58-a352-32c14c9bf18e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:14:30.651465 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4400b329-0424-4d58-a352-32c14c9bf18e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:14:30.651890 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4400b329-0424-4d58-a352-32c14c9bf18e admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:14:30.652062 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4400b329-0424-4d58-a352-32c14c9bf18e admin admin] PATH_INFO: `/v3/users/30321b42749447ea81c84d7d3c8f2094` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:14:30.653230 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4400b329-0424-4d58-a352-32c14c9bf18e admin admin] RBAC: Authorizing `identity:delete_user(user_id=30321b42749447ea81c84d7d3c8f2094)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:14:30.655170 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4400b329-0424-4d58-a352-32c14c9bf18e admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:14:30.813883 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-4400b329-0424-4d58-a352-32c14c9bf18e admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '30321b42749447ea81c84d7d3c8f2094'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:14:30.814546 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-4400b329-0424-4d58-a352-32c14c9bf18e admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '30321b42749447ea81c84d7d3c8f2094'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:14:30.886556 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-4400b329-0424-4d58-a352-32c14c9bf18e admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '30321b42749447ea81c84d7d3c8f2094'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:14:30.887172 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-4400b329-0424-4d58-a352-32c14c9bf18e admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '30321b42749447ea81c84d7d3c8f2094'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:14:30.894888 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-4400b329-0424-4d58-a352-32c14c9bf18e admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '30321b42749447ea81c84d7d3c8f2094'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:14:30.912563 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1438/2877] 10.222.0.27 () {58 vars in 1224 bytes} [Tue Aug 30 02:14:30 2022] DELETE /identity/v3/users/30321b42749447ea81c84d7d3c8f2094 => generated 0 bytes in 331 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 02:14:30.917907 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d73c5101-a3f2-4131-85cb-40f9b83e4268 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:14:30.918387 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d73c5101-a3f2-4131-85cb-40f9b83e4268 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:14:30.918490 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d73c5101-a3f2-4131-85cb-40f9b83e4268 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:14:30.937224 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-d73c5101-a3f2-4131-85cb-40f9b83e4268 None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:14:30.938451 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-d73c5101-a3f2-4131-85cb-40f9b83e4268 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:14:30.996559 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1440/2878] 10.222.0.27 () {58 vars in 958 bytes} [Tue Aug 30 02:14:30 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 80 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 02:14:31.006420 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4400b329-0424-4d58-a352-32c14c9bf18e admin admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:14:31.007832 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-4400b329-0424-4d58-a352-32c14c9bf18e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:14:31.066012 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4400b329-0424-4d58-a352-32c14c9bf18e admin admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:14:31.069717 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e106afd-118e-417f-affd-9ecc616ef291 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f3ea1580c56f4814a92bbb2acf542905', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4562f3951fbe465280073e60a11e44e2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:14:31.070318 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e106afd-118e-417f-affd-9ecc616ef291 service neutron] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:14:31.070598 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e106afd-118e-417f-affd-9ecc616ef291 service neutron] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:14:31.070869 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e106afd-118e-417f-affd-9ecc616ef291 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:14:31.080470 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e106afd-118e-417f-affd-9ecc616ef291 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:14:31.081925 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e106afd-118e-417f-affd-9ecc616ef291 service neutron] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:14:31.091754 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1439/2879] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Aug 30 02:14:31 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 86 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 02:14:31.547427 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d73c5101-a3f2-4131-85cb-40f9b83e4268 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:14:31.548879 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-d73c5101-a3f2-4131-85cb-40f9b83e4268 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:14:31.562679 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d73c5101-a3f2-4131-85cb-40f9b83e4268 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:14:31.567561 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84f9e95b-6190-4bd3-b89b-80b3fd05f029 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:14:31.568696 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84f9e95b-6190-4bd3-b89b-80b3fd05f029 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:14:31.569204 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84f9e95b-6190-4bd3-b89b-80b3fd05f029 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:14:31.569490 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84f9e95b-6190-4bd3-b89b-80b3fd05f029 admin admin] PATH_INFO: `/v3/projects/fc9febd94ffd4099b38f052fbac58372` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:14:31.570945 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84f9e95b-6190-4bd3-b89b-80b3fd05f029 admin admin] RBAC: Authorizing `identity:delete_project(project_id=fc9febd94ffd4099b38f052fbac58372)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:14:31.572873 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84f9e95b-6190-4bd3-b89b-80b3fd05f029 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:14:31.675311 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-84f9e95b-6190-4bd3-b89b-80b3fd05f029 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'fc9febd94ffd4099b38f052fbac58372', 'request_id': 'req-84f9e95b-6190-4bd3-b89b-80b3fd05f029'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:14:31.680611 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-84f9e95b-6190-4bd3-b89b-80b3fd05f029 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'fc9febd94ffd4099b38f052fbac58372', 'request_id': 'req-84f9e95b-6190-4bd3-b89b-80b3fd05f029'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:14:31.687649 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-84f9e95b-6190-4bd3-b89b-80b3fd05f029 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'fc9febd94ffd4099b38f052fbac58372', 'request_id': 'req-84f9e95b-6190-4bd3-b89b-80b3fd05f029'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:14:31.773997 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-84f9e95b-6190-4bd3-b89b-80b3fd05f029 admin admin] The token cache is being invalidate because project fc9febd94ffd4099b38f052fbac58372 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81577) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 30 02:14:31.774293 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-84f9e95b-6190-4bd3-b89b-80b3fd05f029 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:14:31.774923 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-84f9e95b-6190-4bd3-b89b-80b3fd05f029 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:14:31.776593 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1441/2880] 10.222.0.27 () {58 vars in 1233 bytes} [Tue Aug 30 02:14:31 2022] DELETE /identity/v3/projects/fc9febd94ffd4099b38f052fbac58372 => generated 0 bytes in 230 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 02:14:31.781794 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a43647d-e5c9-4e03-b242-cb063c3a6573 None None] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:14:31.782079 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a43647d-e5c9-4e03-b242-cb063c3a6573 None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:14:31.782354 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a43647d-e5c9-4e03-b242-cb063c3a6573 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:14:31.801526 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.auth.core [None req-3a43647d-e5c9-4e03-b242-cb063c3a6573 None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81578) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:14:31.802723 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-3a43647d-e5c9-4e03-b242-cb063c3a6573 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:14:31.836141 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:14:31.862088 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1440/2881] 10.222.0.27 () {58 vars in 958 bytes} [Tue Aug 30 02:14:31 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 81 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 02:14:31.870458 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0d7aae8-a232-4c12-9866-85abe41cfb21 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:14:31.870829 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0d7aae8-a232-4c12-9866-85abe41cfb21 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:14:31.871125 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0d7aae8-a232-4c12-9866-85abe41cfb21 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:14:31.890872 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-b0d7aae8-a232-4c12-9866-85abe41cfb21 None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:14:31.892113 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-b0d7aae8-a232-4c12-9866-85abe41cfb21 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:14:31.900410 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1442/2882] 10.222.0.27 () {58 vars in 958 bytes} [Tue Aug 30 02:14:31 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 02:14:31.904763 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a43647d-e5c9-4e03-b242-cb063c3a6573 None None] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:14:31.915082 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a43647d-e5c9-4e03-b242-cb063c3a6573 None None] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:14:31.920322 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0119bf4b-9f16-4943-8bf8-5fa6d8d716d4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:14:31.921029 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0119bf4b-9f16-4943-8bf8-5fa6d8d716d4 admin admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:14:31.921294 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0119bf4b-9f16-4943-8bf8-5fa6d8d716d4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:14:31.921607 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0119bf4b-9f16-4943-8bf8-5fa6d8d716d4 admin admin] PATH_INFO: `/v3/domains` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:14:31.922258 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0119bf4b-9f16-4943-8bf8-5fa6d8d716d4 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81578) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 02:14:31.922563 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0119bf4b-9f16-4943-8bf8-5fa6d8d716d4 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:14:31.924141 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0119bf4b-9f16-4943-8bf8-5fa6d8d716d4 admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:14:31.936290 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1441/2883] 10.222.0.27 () {58 vars in 1153 bytes} [Tue Aug 30 02:14:31 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 02:14:31.940542 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0d7aae8-a232-4c12-9866-85abe41cfb21 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:14:31.949806 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0d7aae8-a232-4c12-9866-85abe41cfb21 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:14:31.954407 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cec98b45-93a9-461e-aa68-b5a60eba67f9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:14:31.954994 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cec98b45-93a9-461e-aa68-b5a60eba67f9 admin admin] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:14:31.955273 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cec98b45-93a9-461e-aa68-b5a60eba67f9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:14:31.955544 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cec98b45-93a9-461e-aa68-b5a60eba67f9 admin admin] PATH_INFO: `/v3/projects` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:14:31.956166 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cec98b45-93a9-461e-aa68-b5a60eba67f9 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:14:31.958085 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cec98b45-93a9-461e-aa68-b5a60eba67f9 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:14:32.037684 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1443/2884] 10.222.0.27 () {60 vars in 1153 bytes} [Tue Aug 30 02:14:31 2022] POST /identity/v3/projects => generated 385 bytes in 98 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 02:14:32.042160 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0119bf4b-9f16-4943-8bf8-5fa6d8d716d4 admin admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:14:32.052305 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0119bf4b-9f16-4943-8bf8-5fa6d8d716d4 admin admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:14:32.108554 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-520e82b8-92ee-4858-9f27-97b4f9e811fa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:14:32.109150 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-520e82b8-92ee-4858-9f27-97b4f9e811fa admin admin] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:14:32.109719 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-520e82b8-92ee-4858-9f27-97b4f9e811fa admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:14:32.109989 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-520e82b8-92ee-4858-9f27-97b4f9e811fa admin admin] PATH_INFO: `/v3/users` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:14:32.110608 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-520e82b8-92ee-4858-9f27-97b4f9e811fa admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:14:32.112501 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-520e82b8-92ee-4858-9f27-97b4f9e811fa admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:14:32.196860 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1442/2885] 10.222.0.27 () {60 vars in 1144 bytes} [Tue Aug 30 02:14:32 2022] POST /identity/v3/users => generated 293 bytes in 155 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 30 02:14:32.201563 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cec98b45-93a9-461e-aa68-b5a60eba67f9 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:14:32.211950 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cec98b45-93a9-461e-aa68-b5a60eba67f9 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:14:32.217473 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6806192f-f22f-4c8d-9004-3d7a17802ff7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:14:32.218203 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6806192f-f22f-4c8d-9004-3d7a17802ff7 admin admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:14:32.218416 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6806192f-f22f-4c8d-9004-3d7a17802ff7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:14:32.218695 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6806192f-f22f-4c8d-9004-3d7a17802ff7 admin admin] PATH_INFO: `/v3/roles` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:14:32.219333 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6806192f-f22f-4c8d-9004-3d7a17802ff7 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:14:32.220937 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6806192f-f22f-4c8d-9004-3d7a17802ff7 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:14:32.231563 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1444/2886] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Aug 30 02:14:32 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 02:14:32.236141 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-520e82b8-92ee-4858-9f27-97b4f9e811fa admin admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:14:32.246967 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-520e82b8-92ee-4858-9f27-97b4f9e811fa admin admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:14:32.251605 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2147935e-947f-45a8-8469-ff60d0636c30 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:14:32.252304 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2147935e-947f-45a8-8469-ff60d0636c30 admin admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:14:32.252588 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2147935e-947f-45a8-8469-ff60d0636c30 admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:14:32.252864 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2147935e-947f-45a8-8469-ff60d0636c30 admin admin] PATH_INFO: `/v3/roles` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:14:32.253533 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2147935e-947f-45a8-8469-ff60d0636c30 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:14:32.255126 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2147935e-947f-45a8-8469-ff60d0636c30 admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:14:32.266858 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1443/2887] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Aug 30 02:14:32 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 02:14:32.270969 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6806192f-f22f-4c8d-9004-3d7a17802ff7 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:14:32.280896 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6806192f-f22f-4c8d-9004-3d7a17802ff7 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:14:32.285476 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ece93a6-1d81-463a-9816-ac41469c2f00 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:14:32.286207 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ece93a6-1d81-463a-9816-ac41469c2f00 admin admin] REQUEST_METHOD: `PUT` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:14:32.286400 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ece93a6-1d81-463a-9816-ac41469c2f00 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:14:32.286672 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ece93a6-1d81-463a-9816-ac41469c2f00 admin admin] PATH_INFO: `/v3/projects/112feaa3ed7a4adcbc284bb57203ca8c/users/57e4d47a341149c7aa2dc1ab89e21b4b/roles/1fb81afcb9f245f3b989add94598a063` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:14:32.289001 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:14:32.301594 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ece93a6-1d81-463a-9816-ac41469c2f00 admin admin] RBAC: Authorizing `identity:create_grant(project_id=112feaa3ed7a4adcbc284bb57203ca8c, user_id=57e4d47a341149c7aa2dc1ab89e21b4b, role_id=1fb81afcb9f245f3b989add94598a063)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:14:32.304265 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ece93a6-1d81-463a-9816-ac41469c2f00 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:14:32.362700 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1445/2888] 10.222.0.27 () {60 vars in 1483 bytes} [Tue Aug 30 02:14:32 2022] PUT /identity/v3/projects/112feaa3ed7a4adcbc284bb57203ca8c/users/57e4d47a341149c7aa2dc1ab89e21b4b/roles/1fb81afcb9f245f3b989add94598a063 => generated 0 bytes in 92 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 02:14:32.374360 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47d399b9-cbf2-4014-b3cb-17cfddbeb8ac None None] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:14:32.374639 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47d399b9-cbf2-4014-b3cb-17cfddbeb8ac None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:14:32.374907 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47d399b9-cbf2-4014-b3cb-17cfddbeb8ac None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:14:32.403132 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.auth.core [None req-47d399b9-cbf2-4014-b3cb-17cfddbeb8ac None None] MFA Rules not processed for user `57e4d47a341149c7aa2dc1ab89e21b4b`. Rule list: `[]` (Enabled: `True`). {{(pid=81578) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:14:32.404328 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-47d399b9-cbf2-4014-b3cb-17cfddbeb8ac None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:14:32.469817 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1444/2889] 10.222.0.27 () {58 vars in 958 bytes} [Tue Aug 30 02:14:32 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 02:14:32.474293 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ece93a6-1d81-463a-9816-ac41469c2f00 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:14:32.535730 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ece93a6-1d81-463a-9816-ac41469c2f00 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:14:32.540293 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8df37288-c307-4a05-8f12-c8ac3872e5a0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:14:32.540886 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8df37288-c307-4a05-8f12-c8ac3872e5a0 admin admin] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:14:32.541165 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8df37288-c307-4a05-8f12-c8ac3872e5a0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:14:32.542043 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8df37288-c307-4a05-8f12-c8ac3872e5a0 admin admin] PATH_INFO: `/v3/projects` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:14:32.542806 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8df37288-c307-4a05-8f12-c8ac3872e5a0 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:14:32.544688 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8df37288-c307-4a05-8f12-c8ac3872e5a0 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:14:32.653606 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1446/2890] 10.222.0.27 () {60 vars in 1153 bytes} [Tue Aug 30 02:14:32 2022] POST /identity/v3/projects => generated 385 bytes in 180 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 02:14:32.657610 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47d399b9-cbf2-4014-b3cb-17cfddbeb8ac None None] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:14:32.668757 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47d399b9-cbf2-4014-b3cb-17cfddbeb8ac None None] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:14:32.702646 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:14:32.726684 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c098b6ef-a29f-42c1-b057-0c4080aa053a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:14:32.727299 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c098b6ef-a29f-42c1-b057-0c4080aa053a admin admin] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:14:32.727605 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c098b6ef-a29f-42c1-b057-0c4080aa053a admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:14:32.727877 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c098b6ef-a29f-42c1-b057-0c4080aa053a admin admin] PATH_INFO: `/v3/users` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:14:32.728563 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c098b6ef-a29f-42c1-b057-0c4080aa053a admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:14:32.730923 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c098b6ef-a29f-42c1-b057-0c4080aa053a admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:14:32.840972 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1445/2891] 10.222.0.27 () {60 vars in 1144 bytes} [Tue Aug 30 02:14:32 2022] POST /identity/v3/users => generated 293 bytes in 184 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 02:14:32.845025 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8df37288-c307-4a05-8f12-c8ac3872e5a0 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:14:32.856158 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8df37288-c307-4a05-8f12-c8ac3872e5a0 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:14:32.860972 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ecab8bf-1005-4c9a-8c44-ffe1a42b6f10 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:14:32.861596 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ecab8bf-1005-4c9a-8c44-ffe1a42b6f10 admin admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:14:32.861877 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ecab8bf-1005-4c9a-8c44-ffe1a42b6f10 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:14:32.862149 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ecab8bf-1005-4c9a-8c44-ffe1a42b6f10 admin admin] PATH_INFO: `/v3/roles` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:14:32.862788 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ecab8bf-1005-4c9a-8c44-ffe1a42b6f10 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:14:32.864373 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ecab8bf-1005-4c9a-8c44-ffe1a42b6f10 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:14:32.875631 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1447/2892] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Aug 30 02:14:32 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 02:14:32.879708 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c098b6ef-a29f-42c1-b057-0c4080aa053a admin admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:14:32.890431 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c098b6ef-a29f-42c1-b057-0c4080aa053a admin admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:14:32.894909 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49de8113-6021-415e-85a5-f4d1562ed27e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:14:32.895492 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49de8113-6021-415e-85a5-f4d1562ed27e admin admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:14:32.895775 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49de8113-6021-415e-85a5-f4d1562ed27e admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:14:32.896043 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49de8113-6021-415e-85a5-f4d1562ed27e admin admin] PATH_INFO: `/v3/roles` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:14:32.896673 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49de8113-6021-415e-85a5-f4d1562ed27e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:14:32.898276 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49de8113-6021-415e-85a5-f4d1562ed27e admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:14:32.908942 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1446/2893] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Aug 30 02:14:32 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 02:14:32.913923 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ecab8bf-1005-4c9a-8c44-ffe1a42b6f10 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:14:32.931291 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ecab8bf-1005-4c9a-8c44-ffe1a42b6f10 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:14:32.935993 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1428a57d-7eb4-4bab-bdbd-e4eb8da69686 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:14:32.936641 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1428a57d-7eb4-4bab-bdbd-e4eb8da69686 admin admin] REQUEST_METHOD: `PUT` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:14:32.936915 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1428a57d-7eb4-4bab-bdbd-e4eb8da69686 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:14:32.937184 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1428a57d-7eb4-4bab-bdbd-e4eb8da69686 admin admin] PATH_INFO: `/v3/projects/5a607896e3a84ea8a660a455e39ed0ad/users/f684955457294a56b84c9d2ee4694091/roles/1fb81afcb9f245f3b989add94598a063` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:14:32.952235 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1428a57d-7eb4-4bab-bdbd-e4eb8da69686 admin admin] RBAC: Authorizing `identity:create_grant(project_id=5a607896e3a84ea8a660a455e39ed0ad, user_id=f684955457294a56b84c9d2ee4694091, role_id=1fb81afcb9f245f3b989add94598a063)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:14:32.954892 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1428a57d-7eb4-4bab-bdbd-e4eb8da69686 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:14:33.013186 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1448/2894] 10.222.0.27 () {60 vars in 1483 bytes} [Tue Aug 30 02:14:32 2022] PUT /identity/v3/projects/5a607896e3a84ea8a660a455e39ed0ad/users/f684955457294a56b84c9d2ee4694091/roles/1fb81afcb9f245f3b989add94598a063 => generated 0 bytes in 100 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 02:14:33.023748 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7372bff-9eb1-4e9e-bd5b-2a5999dd9bd7 None None] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:14:33.028828 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7372bff-9eb1-4e9e-bd5b-2a5999dd9bd7 None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:14:33.028828 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7372bff-9eb1-4e9e-bd5b-2a5999dd9bd7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:14:33.052222 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.auth.core [None req-e7372bff-9eb1-4e9e-bd5b-2a5999dd9bd7 None None] MFA Rules not processed for user `f684955457294a56b84c9d2ee4694091`. Rule list: `[]` (Enabled: `True`). {{(pid=81578) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:14:33.056213 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-e7372bff-9eb1-4e9e-bd5b-2a5999dd9bd7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:14:33.119280 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1447/2895] 10.222.0.27 () {58 vars in 958 bytes} [Tue Aug 30 02:14:33 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 02:14:33.124391 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1428a57d-7eb4-4bab-bdbd-e4eb8da69686 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:14:33.191826 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1428a57d-7eb4-4bab-bdbd-e4eb8da69686 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:14:33.196677 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0295c989-8599-404d-a3f9-6299b54439d8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:14:33.197278 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0295c989-8599-404d-a3f9-6299b54439d8 admin admin] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:14:33.197723 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0295c989-8599-404d-a3f9-6299b54439d8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:14:33.198032 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0295c989-8599-404d-a3f9-6299b54439d8 admin admin] PATH_INFO: `/v3/projects` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:14:33.198738 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0295c989-8599-404d-a3f9-6299b54439d8 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:14:33.200665 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0295c989-8599-404d-a3f9-6299b54439d8 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:14:33.307346 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1449/2896] 10.222.0.27 () {60 vars in 1153 bytes} [Tue Aug 30 02:14:33 2022] POST /identity/v3/projects => generated 383 bytes in 183 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 02:14:33.311459 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7372bff-9eb1-4e9e-bd5b-2a5999dd9bd7 None None] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:14:33.322054 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7372bff-9eb1-4e9e-bd5b-2a5999dd9bd7 None None] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:14:33.379380 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-078c03fb-5fed-4dd0-84f1-d591630f3153 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:14:33.379959 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-078c03fb-5fed-4dd0-84f1-d591630f3153 admin admin] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:14:33.380239 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-078c03fb-5fed-4dd0-84f1-d591630f3153 admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:14:33.380506 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-078c03fb-5fed-4dd0-84f1-d591630f3153 admin admin] PATH_INFO: `/v3/users` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:14:33.381123 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-078c03fb-5fed-4dd0-84f1-d591630f3153 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:14:33.383055 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-078c03fb-5fed-4dd0-84f1-d591630f3153 admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:14:33.454664 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1448/2897] 10.222.0.27 () {60 vars in 1144 bytes} [Tue Aug 30 02:14:33 2022] POST /identity/v3/users => generated 298 bytes in 144 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 02:14:33.458790 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0295c989-8599-404d-a3f9-6299b54439d8 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:14:33.469746 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0295c989-8599-404d-a3f9-6299b54439d8 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:14:33.474404 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25364256-1957-4ce5-8e6a-0552e6270543 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:14:33.474997 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25364256-1957-4ce5-8e6a-0552e6270543 admin admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:14:33.475269 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25364256-1957-4ce5-8e6a-0552e6270543 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:14:33.475537 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25364256-1957-4ce5-8e6a-0552e6270543 admin admin] PATH_INFO: `/v3/roles` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:14:33.476206 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25364256-1957-4ce5-8e6a-0552e6270543 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:14:33.477834 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25364256-1957-4ce5-8e6a-0552e6270543 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:14:33.488577 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1450/2898] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Aug 30 02:14:33 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 02:14:33.492686 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-078c03fb-5fed-4dd0-84f1-d591630f3153 admin admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:14:33.502926 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-078c03fb-5fed-4dd0-84f1-d591630f3153 admin admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:14:33.507573 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e033ffe-ef35-4b38-96f0-609d4295c7b3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:14:33.508217 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e033ffe-ef35-4b38-96f0-609d4295c7b3 admin admin] REQUEST_METHOD: `PUT` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:14:33.508501 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e033ffe-ef35-4b38-96f0-609d4295c7b3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:14:33.508778 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e033ffe-ef35-4b38-96f0-609d4295c7b3 admin admin] PATH_INFO: `/v3/projects/e024610ecf7240318f88201270d4f19e/users/20a42eba046b41d8bf5f6c26b9c49a16/roles/ead5fac86a0747ea8823a82efcea8ed7` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:14:33.523964 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e033ffe-ef35-4b38-96f0-609d4295c7b3 admin admin] RBAC: Authorizing `identity:create_grant(project_id=e024610ecf7240318f88201270d4f19e, user_id=20a42eba046b41d8bf5f6c26b9c49a16, role_id=ead5fac86a0747ea8823a82efcea8ed7)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:14:33.527251 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e033ffe-ef35-4b38-96f0-609d4295c7b3 admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:14:33.585757 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1449/2899] 10.222.0.27 () {60 vars in 1483 bytes} [Tue Aug 30 02:14:33 2022] PUT /identity/v3/projects/e024610ecf7240318f88201270d4f19e/users/20a42eba046b41d8bf5f6c26b9c49a16/roles/ead5fac86a0747ea8823a82efcea8ed7 => generated 0 bytes in 94 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 30 02:14:33.590509 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25364256-1957-4ce5-8e6a-0552e6270543 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:14:33.600486 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25364256-1957-4ce5-8e6a-0552e6270543 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:14:33.604183 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:14:33.656987 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1371791-eb5a-4a37-8c13-e988e54b64b3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:14:33.657612 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1371791-eb5a-4a37-8c13-e988e54b64b3 admin admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:14:33.657894 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1371791-eb5a-4a37-8c13-e988e54b64b3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:14:33.658167 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1371791-eb5a-4a37-8c13-e988e54b64b3 admin admin] PATH_INFO: `/v3/roles` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:14:33.658802 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1371791-eb5a-4a37-8c13-e988e54b64b3 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:14:33.660391 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1371791-eb5a-4a37-8c13-e988e54b64b3 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:14:33.670028 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1451/2900] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Aug 30 02:14:33 2022] GET /identity/v3/roles => generated 1420 bytes in 80 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 02:14:33.674321 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e033ffe-ef35-4b38-96f0-609d4295c7b3 admin admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:14:33.684744 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e033ffe-ef35-4b38-96f0-609d4295c7b3 admin admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:14:33.689361 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0ceffda-463f-4bd4-84ac-cf995b24f118 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:14:33.690032 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0ceffda-463f-4bd4-84ac-cf995b24f118 admin admin] REQUEST_METHOD: `PUT` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:14:33.690315 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0ceffda-463f-4bd4-84ac-cf995b24f118 admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:14:33.690694 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0ceffda-463f-4bd4-84ac-cf995b24f118 admin admin] PATH_INFO: `/v3/projects/e024610ecf7240318f88201270d4f19e/users/20a42eba046b41d8bf5f6c26b9c49a16/roles/ead5fac86a0747ea8823a82efcea8ed7` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:14:33.692739 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0ceffda-463f-4bd4-84ac-cf995b24f118 admin admin] RBAC: Authorizing `identity:create_grant(project_id=e024610ecf7240318f88201270d4f19e, user_id=20a42eba046b41d8bf5f6c26b9c49a16, role_id=ead5fac86a0747ea8823a82efcea8ed7)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:14:33.695441 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0ceffda-463f-4bd4-84ac-cf995b24f118 admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:14:33.699176 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=92,keystone:INSERT=8 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:14:33.706873 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1450/2901] 10.222.0.27 () {60 vars in 1483 bytes} [Tue Aug 30 02:14:33 2022] PUT /identity/v3/projects/e024610ecf7240318f88201270d4f19e/users/20a42eba046b41d8bf5f6c26b9c49a16/roles/ead5fac86a0747ea8823a82efcea8ed7 => generated 0 bytes in 33 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 02:14:33.723897 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec21bb2f-72c7-41b8-89ef-6a40d107e5dd None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:14:33.724189 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec21bb2f-72c7-41b8-89ef-6a40d107e5dd None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:14:33.724468 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec21bb2f-72c7-41b8-89ef-6a40d107e5dd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:14:33.752740 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-ec21bb2f-72c7-41b8-89ef-6a40d107e5dd None None] MFA Rules not processed for user `20a42eba046b41d8bf5f6c26b9c49a16`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:14:33.753974 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-ec21bb2f-72c7-41b8-89ef-6a40d107e5dd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:14:33.821033 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1452/2902] 10.222.0.27 () {58 vars in 958 bytes} [Tue Aug 30 02:14:33 2022] POST /identity/v3/auth/tokens => generated 3327 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 02:14:33.833549 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0ceffda-463f-4bd4-84ac-cf995b24f118 admin admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:14:33.835272 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-e0ceffda-463f-4bd4-84ac-cf995b24f118 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:14:33.898218 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0ceffda-463f-4bd4-84ac-cf995b24f118 admin admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:14:33.901969 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16870d3a-cb0d-457e-b1c8-52927db3eeea service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ab936b3767a8417aa80c8c8f1b84b667', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4562f3951fbe465280073e60a11e44e2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:14:33.902571 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16870d3a-cb0d-457e-b1c8-52927db3eeea service cinder] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:14:33.902850 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16870d3a-cb0d-457e-b1c8-52927db3eeea service cinder] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:14:33.903121 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16870d3a-cb0d-457e-b1c8-52927db3eeea service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:14:33.962905 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16870d3a-cb0d-457e-b1c8-52927db3eeea service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:14:33.964283 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16870d3a-cb0d-457e-b1c8-52927db3eeea service cinder] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:14:33.974605 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1451/2903] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Aug 30 02:14:33 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 141 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 02:14:35.881287 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec21bb2f-72c7-41b8-89ef-6a40d107e5dd None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:14:35.882631 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-ec21bb2f-72c7-41b8-89ef-6a40d107e5dd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:14:35.894380 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec21bb2f-72c7-41b8-89ef-6a40d107e5dd None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:14:35.898012 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a314e276-c58f-4c4e-8b84-7141cdd558b1 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ab936b3767a8417aa80c8c8f1b84b667', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4562f3951fbe465280073e60a11e44e2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:14:35.898604 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a314e276-c58f-4c4e-8b84-7141cdd558b1 service cinder] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:14:35.898879 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a314e276-c58f-4c4e-8b84-7141cdd558b1 service cinder] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:14:35.899150 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a314e276-c58f-4c4e-8b84-7141cdd558b1 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:14:35.955912 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a314e276-c58f-4c4e-8b84-7141cdd558b1 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:14:35.957370 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a314e276-c58f-4c4e-8b84-7141cdd558b1 service cinder] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:14:35.966765 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1453/2904] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Aug 30 02:14:35 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 86 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 02:14:37.509431 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16870d3a-cb0d-457e-b1c8-52927db3eeea service cinder] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:14:37.518750 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16870d3a-cb0d-457e-b1c8-52927db3eeea service cinder] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:14:37.522357 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff21bf47-97be-4b52-8c2a-71d8e46f5154 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ab936b3767a8417aa80c8c8f1b84b667', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4562f3951fbe465280073e60a11e44e2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:14:37.522947 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff21bf47-97be-4b52-8c2a-71d8e46f5154 service cinder] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:14:37.523227 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff21bf47-97be-4b52-8c2a-71d8e46f5154 service cinder] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:14:37.523499 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff21bf47-97be-4b52-8c2a-71d8e46f5154 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:14:37.532338 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff21bf47-97be-4b52-8c2a-71d8e46f5154 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:14:37.533790 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff21bf47-97be-4b52-8c2a-71d8e46f5154 service cinder] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:14:37.543213 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1452/2905] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Aug 30 02:14:37 2022] GET /identity/v3/auth/tokens => generated 3327 bytes in 34 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 02:14:42.291082 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a314e276-c58f-4c4e-8b84-7141cdd558b1 service cinder] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:14:42.320294 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:14:42.353015 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a314e276-c58f-4c4e-8b84-7141cdd558b1 service cinder] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:14:42.357568 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2b772a1-1177-4309-82ab-bb97af805943 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:14:42.358189 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2b772a1-1177-4309-82ab-bb97af805943 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:14:42.358467 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2b772a1-1177-4309-82ab-bb97af805943 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:14:42.358736 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2b772a1-1177-4309-82ab-bb97af805943 admin admin] PATH_INFO: `/v3/users/57e4d47a341149c7aa2dc1ab89e21b4b` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:14:42.360104 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2b772a1-1177-4309-82ab-bb97af805943 admin admin] RBAC: Authorizing `identity:delete_user(user_id=57e4d47a341149c7aa2dc1ab89e21b4b)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:14:42.362042 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2b772a1-1177-4309-82ab-bb97af805943 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:14:42.530855 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-a2b772a1-1177-4309-82ab-bb97af805943 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '57e4d47a341149c7aa2dc1ab89e21b4b'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:14:42.531463 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-a2b772a1-1177-4309-82ab-bb97af805943 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '57e4d47a341149c7aa2dc1ab89e21b4b'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:14:42.532146 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-a2b772a1-1177-4309-82ab-bb97af805943 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '57e4d47a341149c7aa2dc1ab89e21b4b'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:14:42.545850 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-a2b772a1-1177-4309-82ab-bb97af805943 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '57e4d47a341149c7aa2dc1ab89e21b4b'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:14:42.553311 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-a2b772a1-1177-4309-82ab-bb97af805943 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '57e4d47a341149c7aa2dc1ab89e21b4b'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:14:42.619190 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1454/2906] 10.222.0.27 () {58 vars in 1224 bytes} [Tue Aug 30 02:14:42 2022] DELETE /identity/v3/users/57e4d47a341149c7aa2dc1ab89e21b4b => generated 0 bytes in 329 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 02:14:42.623856 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7aa849e-3c4a-48f3-953b-c9f6930c7802 None None] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:14:42.624367 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7aa849e-3c4a-48f3-953b-c9f6930c7802 None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:14:42.624367 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7aa849e-3c4a-48f3-953b-c9f6930c7802 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:14:42.643602 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.auth.core [None req-b7aa849e-3c4a-48f3-953b-c9f6930c7802 None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81578) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:14:42.644788 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-b7aa849e-3c4a-48f3-953b-c9f6930c7802 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:14:42.703280 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1453/2907] 10.222.0.27 () {58 vars in 958 bytes} [Tue Aug 30 02:14:42 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 80 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 02:14:42.713001 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2b772a1-1177-4309-82ab-bb97af805943 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:14:42.714512 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-a2b772a1-1177-4309-82ab-bb97af805943 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:14:42.771734 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2b772a1-1177-4309-82ab-bb97af805943 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:14:42.775390 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-490cd7fb-702e-4350-b0ec-ff62c2839b60 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f3ea1580c56f4814a92bbb2acf542905', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4562f3951fbe465280073e60a11e44e2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:14:42.776023 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-490cd7fb-702e-4350-b0ec-ff62c2839b60 service neutron] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:14:42.776431 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-490cd7fb-702e-4350-b0ec-ff62c2839b60 service neutron] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:14:42.776712 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-490cd7fb-702e-4350-b0ec-ff62c2839b60 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:14:42.785772 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-490cd7fb-702e-4350-b0ec-ff62c2839b60 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:14:42.787132 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-490cd7fb-702e-4350-b0ec-ff62c2839b60 service neutron] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:14:42.796306 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1455/2908] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Aug 30 02:14:42 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 02:14:43.163677 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7aa849e-3c4a-48f3-953b-c9f6930c7802 None None] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:14:43.169287 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-b7aa849e-3c4a-48f3-953b-c9f6930c7802 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:14:43.178032 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7aa849e-3c4a-48f3-953b-c9f6930c7802 None None] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:14:43.183385 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89e89528-1e21-4b79-ad36-6a946b7d80a4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:14:43.184026 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89e89528-1e21-4b79-ad36-6a946b7d80a4 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:14:43.184307 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89e89528-1e21-4b79-ad36-6a946b7d80a4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:14:43.184580 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89e89528-1e21-4b79-ad36-6a946b7d80a4 admin admin] PATH_INFO: `/v3/projects/112feaa3ed7a4adcbc284bb57203ca8c` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:14:43.185748 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89e89528-1e21-4b79-ad36-6a946b7d80a4 admin admin] RBAC: Authorizing `identity:delete_project(project_id=112feaa3ed7a4adcbc284bb57203ca8c)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:14:43.187810 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89e89528-1e21-4b79-ad36-6a946b7d80a4 admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:14:43.286422 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=98,keystone:DELETE=2 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:14:43.298683 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-89e89528-1e21-4b79-ad36-6a946b7d80a4 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '112feaa3ed7a4adcbc284bb57203ca8c', 'request_id': 'req-89e89528-1e21-4b79-ad36-6a946b7d80a4'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:14:43.304116 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-89e89528-1e21-4b79-ad36-6a946b7d80a4 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '112feaa3ed7a4adcbc284bb57203ca8c', 'request_id': 'req-89e89528-1e21-4b79-ad36-6a946b7d80a4'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:14:43.310887 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-89e89528-1e21-4b79-ad36-6a946b7d80a4 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '112feaa3ed7a4adcbc284bb57203ca8c', 'request_id': 'req-89e89528-1e21-4b79-ad36-6a946b7d80a4'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:14:43.389533 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-89e89528-1e21-4b79-ad36-6a946b7d80a4 admin admin] The token cache is being invalidate because project 112feaa3ed7a4adcbc284bb57203ca8c was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81578) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 30 02:14:43.389825 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-89e89528-1e21-4b79-ad36-6a946b7d80a4 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:14:43.390500 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-89e89528-1e21-4b79-ad36-6a946b7d80a4 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:14:43.391992 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1454/2909] 10.222.0.27 () {58 vars in 1233 bytes} [Tue Aug 30 02:14:43 2022] DELETE /identity/v3/projects/112feaa3ed7a4adcbc284bb57203ca8c => generated 0 bytes in 229 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 02:14:43.395988 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-490cd7fb-702e-4350-b0ec-ff62c2839b60 service neutron] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:14:43.405761 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-490cd7fb-702e-4350-b0ec-ff62c2839b60 service neutron] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:14:43.449857 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:14:43.461247 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e5361a1-feb3-41f5-a651-1aa03d5c5d7c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:14:43.461900 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e5361a1-feb3-41f5-a651-1aa03d5c5d7c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:14:43.462181 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e5361a1-feb3-41f5-a651-1aa03d5c5d7c admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:14:43.462460 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e5361a1-feb3-41f5-a651-1aa03d5c5d7c admin admin] PATH_INFO: `/v3/users/f684955457294a56b84c9d2ee4694091` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:14:43.464147 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e5361a1-feb3-41f5-a651-1aa03d5c5d7c admin admin] RBAC: Authorizing `identity:delete_user(user_id=f684955457294a56b84c9d2ee4694091)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:14:43.466095 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e5361a1-feb3-41f5-a651-1aa03d5c5d7c admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:14:43.644915 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-0e5361a1-feb3-41f5-a651-1aa03d5c5d7c admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'f684955457294a56b84c9d2ee4694091'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:14:43.645533 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-0e5361a1-feb3-41f5-a651-1aa03d5c5d7c admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'f684955457294a56b84c9d2ee4694091'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:14:43.646134 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-0e5361a1-feb3-41f5-a651-1aa03d5c5d7c admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'f684955457294a56b84c9d2ee4694091'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:14:43.657910 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-0e5361a1-feb3-41f5-a651-1aa03d5c5d7c admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'f684955457294a56b84c9d2ee4694091'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:14:43.664635 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-0e5361a1-feb3-41f5-a651-1aa03d5c5d7c admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'f684955457294a56b84c9d2ee4694091'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:14:43.734098 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1456/2910] 10.222.0.27 () {58 vars in 1224 bytes} [Tue Aug 30 02:14:43 2022] DELETE /identity/v3/users/f684955457294a56b84c9d2ee4694091 => generated 0 bytes in 339 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 02:14:44.156905 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89e89528-1e21-4b79-ad36-6a946b7d80a4 admin admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:14:44.158491 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-89e89528-1e21-4b79-ad36-6a946b7d80a4 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:14:44.171711 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89e89528-1e21-4b79-ad36-6a946b7d80a4 admin admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:14:44.230655 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0804c02a-cbde-4d7d-b7fb-13505d62c3d6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:14:44.231286 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0804c02a-cbde-4d7d-b7fb-13505d62c3d6 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:14:44.231567 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0804c02a-cbde-4d7d-b7fb-13505d62c3d6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:14:44.231838 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0804c02a-cbde-4d7d-b7fb-13505d62c3d6 admin admin] PATH_INFO: `/v3/projects/5a607896e3a84ea8a660a455e39ed0ad` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:14:44.232949 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0804c02a-cbde-4d7d-b7fb-13505d62c3d6 admin admin] RBAC: Authorizing `identity:delete_project(project_id=5a607896e3a84ea8a660a455e39ed0ad)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:14:44.235144 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0804c02a-cbde-4d7d-b7fb-13505d62c3d6 admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:14:44.345365 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-0804c02a-cbde-4d7d-b7fb-13505d62c3d6 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '5a607896e3a84ea8a660a455e39ed0ad', 'request_id': 'req-0804c02a-cbde-4d7d-b7fb-13505d62c3d6'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:14:44.351010 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-0804c02a-cbde-4d7d-b7fb-13505d62c3d6 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '5a607896e3a84ea8a660a455e39ed0ad', 'request_id': 'req-0804c02a-cbde-4d7d-b7fb-13505d62c3d6'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:14:44.358012 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-0804c02a-cbde-4d7d-b7fb-13505d62c3d6 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '5a607896e3a84ea8a660a455e39ed0ad', 'request_id': 'req-0804c02a-cbde-4d7d-b7fb-13505d62c3d6'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:14:44.424517 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-0804c02a-cbde-4d7d-b7fb-13505d62c3d6 admin admin] The token cache is being invalidate because project 5a607896e3a84ea8a660a455e39ed0ad was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81578) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 30 02:14:44.424735 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-0804c02a-cbde-4d7d-b7fb-13505d62c3d6 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:14:44.425373 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-0804c02a-cbde-4d7d-b7fb-13505d62c3d6 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:14:44.426904 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1455/2911] 10.222.0.27 () {58 vars in 1233 bytes} [Tue Aug 30 02:14:44 2022] DELETE /identity/v3/projects/5a607896e3a84ea8a660a455e39ed0ad => generated 0 bytes in 271 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 02:14:44.430903 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e5361a1-feb3-41f5-a651-1aa03d5c5d7c admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:14:44.432310 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-0e5361a1-feb3-41f5-a651-1aa03d5c5d7c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:14:44.445345 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e5361a1-feb3-41f5-a651-1aa03d5c5d7c admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:14:44.500707 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-defc38d6-be9a-4ade-86df-f57a8fce10d0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:14:44.501326 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-defc38d6-be9a-4ade-86df-f57a8fce10d0 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:14:44.501637 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-defc38d6-be9a-4ade-86df-f57a8fce10d0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:14:44.501909 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-defc38d6-be9a-4ade-86df-f57a8fce10d0 admin admin] PATH_INFO: `/v3/users/20a42eba046b41d8bf5f6c26b9c49a16` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:14:44.503000 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-defc38d6-be9a-4ade-86df-f57a8fce10d0 admin admin] RBAC: Authorizing `identity:delete_user(user_id=20a42eba046b41d8bf5f6c26b9c49a16)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:14:44.504881 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-defc38d6-be9a-4ade-86df-f57a8fce10d0 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:14:44.702496 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-defc38d6-be9a-4ade-86df-f57a8fce10d0 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '20a42eba046b41d8bf5f6c26b9c49a16'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:14:44.706705 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-defc38d6-be9a-4ade-86df-f57a8fce10d0 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '20a42eba046b41d8bf5f6c26b9c49a16'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:14:44.707331 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-defc38d6-be9a-4ade-86df-f57a8fce10d0 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '20a42eba046b41d8bf5f6c26b9c49a16'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:14:44.720254 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-defc38d6-be9a-4ade-86df-f57a8fce10d0 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '20a42eba046b41d8bf5f6c26b9c49a16'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:14:44.724782 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=80,keystone:DELETE=19,keystone:INSERT=1 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:14:44.727663 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-defc38d6-be9a-4ade-86df-f57a8fce10d0 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '20a42eba046b41d8bf5f6c26b9c49a16'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:14:44.801385 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1457/2912] 10.222.0.27 () {58 vars in 1224 bytes} [Tue Aug 30 02:14:44 2022] DELETE /identity/v3/users/20a42eba046b41d8bf5f6c26b9c49a16 => generated 0 bytes in 371 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 02:14:45.255860 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0804c02a-cbde-4d7d-b7fb-13505d62c3d6 admin admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:14:45.258613 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-0804c02a-cbde-4d7d-b7fb-13505d62c3d6 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:14:45.272083 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0804c02a-cbde-4d7d-b7fb-13505d62c3d6 admin admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:14:45.320615 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=86,keystone:DELETE=12,keystone:INSERT=2 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:14:45.330964 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-820f6e35-efda-45f4-b6e8-f7c0ff7401b3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:14:45.331589 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-820f6e35-efda-45f4-b6e8-f7c0ff7401b3 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:14:45.331864 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-820f6e35-efda-45f4-b6e8-f7c0ff7401b3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:14:45.332132 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-820f6e35-efda-45f4-b6e8-f7c0ff7401b3 admin admin] PATH_INFO: `/v3/projects/e024610ecf7240318f88201270d4f19e` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:14:45.333275 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-820f6e35-efda-45f4-b6e8-f7c0ff7401b3 admin admin] RBAC: Authorizing `identity:delete_project(project_id=e024610ecf7240318f88201270d4f19e)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:14:45.335183 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-820f6e35-efda-45f4-b6e8-f7c0ff7401b3 admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:14:45.466760 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-820f6e35-efda-45f4-b6e8-f7c0ff7401b3 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'e024610ecf7240318f88201270d4f19e', 'request_id': 'req-820f6e35-efda-45f4-b6e8-f7c0ff7401b3'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:14:45.472206 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-820f6e35-efda-45f4-b6e8-f7c0ff7401b3 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'e024610ecf7240318f88201270d4f19e', 'request_id': 'req-820f6e35-efda-45f4-b6e8-f7c0ff7401b3'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:14:45.478255 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-820f6e35-efda-45f4-b6e8-f7c0ff7401b3 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'e024610ecf7240318f88201270d4f19e', 'request_id': 'req-820f6e35-efda-45f4-b6e8-f7c0ff7401b3'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:14:45.573108 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-820f6e35-efda-45f4-b6e8-f7c0ff7401b3 admin admin] The token cache is being invalidate because project e024610ecf7240318f88201270d4f19e was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81578) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 30 02:14:45.573397 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-820f6e35-efda-45f4-b6e8-f7c0ff7401b3 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:14:45.574044 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-820f6e35-efda-45f4-b6e8-f7c0ff7401b3 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:14:45.575539 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1456/2913] 10.222.0.27 () {58 vars in 1233 bytes} [Tue Aug 30 02:14:45 2022] DELETE /identity/v3/projects/e024610ecf7240318f88201270d4f19e => generated 0 bytes in 320 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 02:14:45.580426 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-106fc074-2ccf-4e7b-b24c-f50d90b9c64f None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:14:45.580920 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-106fc074-2ccf-4e7b-b24c-f50d90b9c64f None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:14:45.581045 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-106fc074-2ccf-4e7b-b24c-f50d90b9c64f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:14:45.601071 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-106fc074-2ccf-4e7b-b24c-f50d90b9c64f None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:14:45.603901 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-106fc074-2ccf-4e7b-b24c-f50d90b9c64f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:14:45.661560 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1458/2914] 10.222.0.27 () {58 vars in 958 bytes} [Tue Aug 30 02:14:45 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 82 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 02:14:45.670243 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c35e5efc-8d3b-42cd-8a49-a73003b182fe None None] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:14:45.670533 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c35e5efc-8d3b-42cd-8a49-a73003b182fe None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:14:45.670840 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c35e5efc-8d3b-42cd-8a49-a73003b182fe None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:14:45.690397 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.auth.core [None req-c35e5efc-8d3b-42cd-8a49-a73003b182fe None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81578) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:14:45.691605 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-c35e5efc-8d3b-42cd-8a49-a73003b182fe None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:14:45.700069 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1457/2915] 10.222.0.27 () {58 vars in 958 bytes} [Tue Aug 30 02:14:45 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 30 02:14:45.704374 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-106fc074-2ccf-4e7b-b24c-f50d90b9c64f None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:14:45.714302 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-106fc074-2ccf-4e7b-b24c-f50d90b9c64f None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:14:45.719321 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d578db5-9c8e-4abe-926e-a047aa06a916 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:14:45.719932 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d578db5-9c8e-4abe-926e-a047aa06a916 admin admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:14:45.720314 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d578db5-9c8e-4abe-926e-a047aa06a916 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:14:45.720604 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d578db5-9c8e-4abe-926e-a047aa06a916 admin admin] PATH_INFO: `/v3/domains` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:14:45.721324 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d578db5-9c8e-4abe-926e-a047aa06a916 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 30 02:14:45.721653 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d578db5-9c8e-4abe-926e-a047aa06a916 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:14:45.723243 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d578db5-9c8e-4abe-926e-a047aa06a916 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:14:45.736146 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1459/2916] 10.222.0.27 () {58 vars in 1153 bytes} [Tue Aug 30 02:14:45 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 30 02:14:45.740428 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c35e5efc-8d3b-42cd-8a49-a73003b182fe None None] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:14:45.749654 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c35e5efc-8d3b-42cd-8a49-a73003b182fe None None] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:14:45.754470 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ce94b18-04fb-4bea-8141-5bc734452cc8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:14:45.755059 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ce94b18-04fb-4bea-8141-5bc734452cc8 admin admin] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:14:45.755337 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ce94b18-04fb-4bea-8141-5bc734452cc8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:14:45.755606 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ce94b18-04fb-4bea-8141-5bc734452cc8 admin admin] PATH_INFO: `/v3/projects` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:14:45.756237 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ce94b18-04fb-4bea-8141-5bc734452cc8 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:14:45.758150 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ce94b18-04fb-4bea-8141-5bc734452cc8 admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:14:45.878016 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1458/2917] 10.222.0.27 () {60 vars in 1153 bytes} [Tue Aug 30 02:14:45 2022] POST /identity/v3/projects => generated 397 bytes in 138 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 02:14:45.882130 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d578db5-9c8e-4abe-926e-a047aa06a916 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:14:45.892459 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d578db5-9c8e-4abe-926e-a047aa06a916 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:14:45.952039 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cb48dd8-61aa-47e7-8201-fd29dede91b0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:14:45.952636 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cb48dd8-61aa-47e7-8201-fd29dede91b0 admin admin] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:14:45.952914 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cb48dd8-61aa-47e7-8201-fd29dede91b0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:14:45.953182 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cb48dd8-61aa-47e7-8201-fd29dede91b0 admin admin] PATH_INFO: `/v3/users` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:14:45.953882 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8cb48dd8-61aa-47e7-8201-fd29dede91b0 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:14:45.955785 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8cb48dd8-61aa-47e7-8201-fd29dede91b0 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:14:46.065971 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1460/2918] 10.222.0.27 () {60 vars in 1144 bytes} [Tue Aug 30 02:14:45 2022] POST /identity/v3/users => generated 299 bytes in 184 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 30 02:14:46.070066 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ce94b18-04fb-4bea-8141-5bc734452cc8 admin admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:14:46.080908 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ce94b18-04fb-4bea-8141-5bc734452cc8 admin admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:14:46.085546 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-834f7cd8-7a94-4b76-8890-f45011d69676 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:14:46.086197 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-834f7cd8-7a94-4b76-8890-f45011d69676 admin admin] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:14:46.086480 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-834f7cd8-7a94-4b76-8890-f45011d69676 admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:14:46.086754 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-834f7cd8-7a94-4b76-8890-f45011d69676 admin admin] PATH_INFO: `/v3/roles` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:14:46.087424 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-834f7cd8-7a94-4b76-8890-f45011d69676 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:14:46.089060 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-834f7cd8-7a94-4b76-8890-f45011d69676 admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:14:46.099132 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1459/2919] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Aug 30 02:14:46 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 30 02:14:46.103028 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cb48dd8-61aa-47e7-8201-fd29dede91b0 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:14:46.113888 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cb48dd8-61aa-47e7-8201-fd29dede91b0 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:14:46.118585 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1164cc44-07ed-4019-96b0-8d028ae7d71f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:14:46.119178 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1164cc44-07ed-4019-96b0-8d028ae7d71f admin admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:14:46.119459 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1164cc44-07ed-4019-96b0-8d028ae7d71f admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:14:46.119728 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1164cc44-07ed-4019-96b0-8d028ae7d71f admin admin] PATH_INFO: `/v3/roles` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:14:46.120399 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1164cc44-07ed-4019-96b0-8d028ae7d71f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:14:46.122635 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1164cc44-07ed-4019-96b0-8d028ae7d71f admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:14:46.133371 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1461/2920] 10.222.0.27 () {58 vars in 1122 bytes} [Tue Aug 30 02:14:46 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 30 02:14:46.137654 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-834f7cd8-7a94-4b76-8890-f45011d69676 admin admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:14:46.147765 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-834f7cd8-7a94-4b76-8890-f45011d69676 admin admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:14:46.152292 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c03c7f4-72fe-438d-b56d-6e4c99f4a373 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:14:46.153116 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c03c7f4-72fe-438d-b56d-6e4c99f4a373 admin admin] REQUEST_METHOD: `PUT` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:14:46.153429 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c03c7f4-72fe-438d-b56d-6e4c99f4a373 admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:14:46.153737 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c03c7f4-72fe-438d-b56d-6e4c99f4a373 admin admin] PATH_INFO: `/v3/projects/a9803630c7d44fa8b8d2a944e4af4a06/users/4886efe163eb4f959c0d2453b1f5eec2/roles/1fb81afcb9f245f3b989add94598a063` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:14:46.168764 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c03c7f4-72fe-438d-b56d-6e4c99f4a373 admin admin] RBAC: Authorizing `identity:create_grant(project_id=a9803630c7d44fa8b8d2a944e4af4a06, user_id=4886efe163eb4f959c0d2453b1f5eec2, role_id=1fb81afcb9f245f3b989add94598a063)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:14:46.171527 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c03c7f4-72fe-438d-b56d-6e4c99f4a373 admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:14:46.240230 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1460/2921] 10.222.0.27 () {60 vars in 1483 bytes} [Tue Aug 30 02:14:46 2022] PUT /identity/v3/projects/a9803630c7d44fa8b8d2a944e4af4a06/users/4886efe163eb4f959c0d2453b1f5eec2/roles/1fb81afcb9f245f3b989add94598a063 => generated 0 bytes in 103 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 02:14:46.251235 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98f55948-9cc5-44d5-ac94-0b5c2bbfd7b2 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:14:46.251551 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98f55948-9cc5-44d5-ac94-0b5c2bbfd7b2 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:14:46.251826 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98f55948-9cc5-44d5-ac94-0b5c2bbfd7b2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:14:46.285489 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-98f55948-9cc5-44d5-ac94-0b5c2bbfd7b2 None None] MFA Rules not processed for user `4886efe163eb4f959c0d2453b1f5eec2`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:14:46.286717 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-98f55948-9cc5-44d5-ac94-0b5c2bbfd7b2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:14:46.329668 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=95,keystone:INSERT=4,keystone:DELETE=1 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:14:46.351616 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1462/2922] 10.222.0.27 () {58 vars in 958 bytes} [Tue Aug 30 02:14:46 2022] POST /identity/v3/auth/tokens => generated 3274 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 02:14:46.364107 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c03c7f4-72fe-438d-b56d-6e4c99f4a373 admin admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:14:46.365903 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-7c03c7f4-72fe-438d-b56d-6e4c99f4a373 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:14:46.423830 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c03c7f4-72fe-438d-b56d-6e4c99f4a373 admin admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:14:46.427357 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9b77815-fd1b-4457-a44e-c7b4d3db504e service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ab936b3767a8417aa80c8c8f1b84b667', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4562f3951fbe465280073e60a11e44e2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:14:46.427955 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9b77815-fd1b-4457-a44e-c7b4d3db504e service cinder] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:14:46.428256 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9b77815-fd1b-4457-a44e-c7b4d3db504e service cinder] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:14:46.428523 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9b77815-fd1b-4457-a44e-c7b4d3db504e service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:14:46.437439 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9b77815-fd1b-4457-a44e-c7b4d3db504e service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:14:46.438862 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9b77815-fd1b-4457-a44e-c7b4d3db504e service cinder] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:14:46.447406 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1461/2923] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Aug 30 02:14:46 2022] GET /identity/v3/auth/tokens => generated 3274 bytes in 84 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 02:14:55.309319 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98f55948-9cc5-44d5-ac94-0b5c2bbfd7b2 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:14:55.371966 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98f55948-9cc5-44d5-ac94-0b5c2bbfd7b2 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:14:55.376491 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e43365f-b9a9-4eb0-99f2-554f3d8a4b14 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:14:55.377119 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e43365f-b9a9-4eb0-99f2-554f3d8a4b14 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:14:55.377399 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e43365f-b9a9-4eb0-99f2-554f3d8a4b14 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:14:55.377699 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e43365f-b9a9-4eb0-99f2-554f3d8a4b14 admin admin] PATH_INFO: `/v3/users/4886efe163eb4f959c0d2453b1f5eec2` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:14:55.378812 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e43365f-b9a9-4eb0-99f2-554f3d8a4b14 admin admin] RBAC: Authorizing `identity:delete_user(user_id=4886efe163eb4f959c0d2453b1f5eec2)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:14:55.380733 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e43365f-b9a9-4eb0-99f2-554f3d8a4b14 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:14:55.547315 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-2e43365f-b9a9-4eb0-99f2-554f3d8a4b14 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '4886efe163eb4f959c0d2453b1f5eec2'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:14:55.547965 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-2e43365f-b9a9-4eb0-99f2-554f3d8a4b14 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '4886efe163eb4f959c0d2453b1f5eec2'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:14:55.548584 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-2e43365f-b9a9-4eb0-99f2-554f3d8a4b14 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '4886efe163eb4f959c0d2453b1f5eec2'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:14:55.560161 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-2e43365f-b9a9-4eb0-99f2-554f3d8a4b14 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '4886efe163eb4f959c0d2453b1f5eec2'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:14:55.566860 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-2e43365f-b9a9-4eb0-99f2-554f3d8a4b14 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '4886efe163eb4f959c0d2453b1f5eec2'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:14:55.630294 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1463/2924] 10.222.0.27 () {58 vars in 1224 bytes} [Tue Aug 30 02:14:55 2022] DELETE /identity/v3/users/4886efe163eb4f959c0d2453b1f5eec2 => generated 0 bytes in 322 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 02:14:55.635163 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee778eb7-1ae4-4957-884a-d5090f0fdc5b None None] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:14:55.635626 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee778eb7-1ae4-4957-884a-d5090f0fdc5b None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:14:55.636521 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee778eb7-1ae4-4957-884a-d5090f0fdc5b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:14:55.658163 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.auth.core [None req-ee778eb7-1ae4-4957-884a-d5090f0fdc5b None None] MFA Rules not processed for user `ac83d5ed111c4f4e8b3c3226bd3bbaba`. Rule list: `[]` (Enabled: `True`). {{(pid=81578) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:14:55.659407 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-ee778eb7-1ae4-4957-884a-d5090f0fdc5b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:14:55.673500 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:14:55.722409 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1462/2925] 10.222.0.27 () {58 vars in 958 bytes} [Tue Aug 30 02:14:55 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 02:14:55.732926 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e43365f-b9a9-4eb0-99f2-554f3d8a4b14 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:14:55.734339 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-2e43365f-b9a9-4eb0-99f2-554f3d8a4b14 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:14:55.791367 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e43365f-b9a9-4eb0-99f2-554f3d8a4b14 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:14:55.795817 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-057f780f-32b0-4811-9bb5-c20a97ea2c06 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f3ea1580c56f4814a92bbb2acf542905', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4562f3951fbe465280073e60a11e44e2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:14:55.796416 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-057f780f-32b0-4811-9bb5-c20a97ea2c06 service neutron] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:14:55.796693 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-057f780f-32b0-4811-9bb5-c20a97ea2c06 service neutron] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:14:55.796964 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-057f780f-32b0-4811-9bb5-c20a97ea2c06 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:14:55.805884 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-057f780f-32b0-4811-9bb5-c20a97ea2c06 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:14:55.807262 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-057f780f-32b0-4811-9bb5-c20a97ea2c06 service neutron] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:14:55.817046 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1464/2926] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Aug 30 02:14:55 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 85 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 02:14:56.230898 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee778eb7-1ae4-4957-884a-d5090f0fdc5b None None] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:14:56.232309 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-ee778eb7-1ae4-4957-884a-d5090f0fdc5b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:14:56.244894 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee778eb7-1ae4-4957-884a-d5090f0fdc5b None None] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:14:56.265585 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8cbd9c4-2bab-4bdc-b3dc-569f31e047e7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:14:56.266259 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8cbd9c4-2bab-4bdc-b3dc-569f31e047e7 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:14:56.266542 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8cbd9c4-2bab-4bdc-b3dc-569f31e047e7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:14:56.266817 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8cbd9c4-2bab-4bdc-b3dc-569f31e047e7 admin admin] PATH_INFO: `/v3/projects/a9803630c7d44fa8b8d2a944e4af4a06` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:14:56.268020 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8cbd9c4-2bab-4bdc-b3dc-569f31e047e7 admin admin] RBAC: Authorizing `identity:delete_project(project_id=a9803630c7d44fa8b8d2a944e4af4a06)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:14:56.269973 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8cbd9c4-2bab-4bdc-b3dc-569f31e047e7 admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:14:56.379210 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-f8cbd9c4-2bab-4bdc-b3dc-569f31e047e7 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'a9803630c7d44fa8b8d2a944e4af4a06', 'request_id': 'req-f8cbd9c4-2bab-4bdc-b3dc-569f31e047e7'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:14:56.384558 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-f8cbd9c4-2bab-4bdc-b3dc-569f31e047e7 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'a9803630c7d44fa8b8d2a944e4af4a06', 'request_id': 'req-f8cbd9c4-2bab-4bdc-b3dc-569f31e047e7'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:14:56.391817 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-f8cbd9c4-2bab-4bdc-b3dc-569f31e047e7 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'a9803630c7d44fa8b8d2a944e4af4a06', 'request_id': 'req-f8cbd9c4-2bab-4bdc-b3dc-569f31e047e7'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:14:56.464327 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-f8cbd9c4-2bab-4bdc-b3dc-569f31e047e7 admin admin] The token cache is being invalidate because project a9803630c7d44fa8b8d2a944e4af4a06 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81578) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 30 02:14:56.464731 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-f8cbd9c4-2bab-4bdc-b3dc-569f31e047e7 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:14:56.465249 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-f8cbd9c4-2bab-4bdc-b3dc-569f31e047e7 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:14:56.466763 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1463/2927] 10.222.0.27 () {58 vars in 1233 bytes} [Tue Aug 30 02:14:56 2022] DELETE /identity/v3/projects/a9803630c7d44fa8b8d2a944e4af4a06 => generated 0 bytes in 237 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 02:15:05.812859 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=87,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:15:06.461197 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=45,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:15:19.276570 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-057f780f-32b0-4811-9bb5-c20a97ea2c06 service neutron] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:15:19.278101 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-057f780f-32b0-4811-9bb5-c20a97ea2c06 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:15:19.347088 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-057f780f-32b0-4811-9bb5-c20a97ea2c06 service neutron] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:15:19.405340 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7f6a09c-e457-4785-aba6-454311471d29 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '379536ff06fd4f719899e5dde732e63d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4562f3951fbe465280073e60a11e44e2', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:15:19.405974 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7f6a09c-e457-4785-aba6-454311471d29 service placement] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:15:19.406256 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7f6a09c-e457-4785-aba6-454311471d29 service placement] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:15:19.406532 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7f6a09c-e457-4785-aba6-454311471d29 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:15:19.408258 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-c7f6a09c-e457-4785-aba6-454311471d29 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:15:19.419713 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7f6a09c-e457-4785-aba6-454311471d29 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:15:19.421272 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7f6a09c-e457-4785-aba6-454311471d29 service placement] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:15:19.430902 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1465/2928] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Aug 30 02:15:19 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 155 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 02:15:29.426790 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=58 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:15:54.117386 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68ad5f8f-002d-4d54-92dc-e786af920036 None None] REQUEST_METHOD: `POST` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:15:54.117950 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68ad5f8f-002d-4d54-92dc-e786af920036 None None] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:15:54.117950 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68ad5f8f-002d-4d54-92dc-e786af920036 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:15:54.137120 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.auth.core [None req-68ad5f8f-002d-4d54-92dc-e786af920036 None None] MFA Rules not processed for user `40e0086c1b924e7f86bd814bb45be3d9`. Rule list: `[]` (Enabled: `True`). {{(pid=81578) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 30 02:15:54.138332 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-68ad5f8f-002d-4d54-92dc-e786af920036 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:15:54.193876 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1464/2929] 10.222.0.27 () {58 vars in 958 bytes} [Tue Aug 30 02:15:54 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 77 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 30 02:15:54.521102 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7f6a09c-e457-4785-aba6-454311471d29 service placement] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:15:54.522465 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-c7f6a09c-e457-4785-aba6-454311471d29 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:15:54.577811 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7f6a09c-e457-4785-aba6-454311471d29 service placement] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:15:54.581422 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-464e9d8d-d92e-4085-a86a-a0c6c9a83f28 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f3ea1580c56f4814a92bbb2acf542905', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4562f3951fbe465280073e60a11e44e2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:15:54.582043 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-464e9d8d-d92e-4085-a86a-a0c6c9a83f28 service neutron] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:15:54.582323 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-464e9d8d-d92e-4085-a86a-a0c6c9a83f28 service neutron] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:15:54.582593 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-464e9d8d-d92e-4085-a86a-a0c6c9a83f28 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:15:54.584295 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-464e9d8d-d92e-4085-a86a-a0c6c9a83f28 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:15:54.594208 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-464e9d8d-d92e-4085-a86a-a0c6c9a83f28 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:15:54.595615 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-464e9d8d-d92e-4085-a86a-a0c6c9a83f28 service neutron] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:15:54.604064 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1466/2930] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Aug 30 02:15:54 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 84 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 02:16:04.187039 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=25 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:16:04.599847 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=26 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:16:06.791567 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68ad5f8f-002d-4d54-92dc-e786af920036 None None] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:16:06.792959 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-68ad5f8f-002d-4d54-92dc-e786af920036 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:16:06.868608 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68ad5f8f-002d-4d54-92dc-e786af920036 None None] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:16:06.874330 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0dce2aec-e339-4a7a-84d8-3aff288851a2 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd64f2e71361644589bec8a06e6eceed1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4562f3951fbe465280073e60a11e44e2', 'project_domain_id': 'default', 'roles': ['member', 'service', 'admin', 'ResellerAdmin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:16:06.874939 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dce2aec-e339-4a7a-84d8-3aff288851a2 service nova] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:16:06.875216 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dce2aec-e339-4a7a-84d8-3aff288851a2 service nova] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:16:06.875486 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dce2aec-e339-4a7a-84d8-3aff288851a2 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:16:06.877377 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-0dce2aec-e339-4a7a-84d8-3aff288851a2 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:16:06.888610 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0dce2aec-e339-4a7a-84d8-3aff288851a2 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:16:06.890231 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0dce2aec-e339-4a7a-84d8-3aff288851a2 service nova] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:16:06.899284 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1465/2931] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Aug 30 02:16:06 2022] GET /identity/v3/auth/tokens => generated 3256 bytes in 108 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 02:16:16.895131 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=34 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:17:54.030369 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-464e9d8d-d92e-4085-a86a-a0c6c9a83f28 service neutron] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:17:54.032051 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-464e9d8d-d92e-4085-a86a-a0c6c9a83f28 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:17:54.043667 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-464e9d8d-d92e-4085-a86a-a0c6c9a83f28 service neutron] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:17:54.048087 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f95e12c-517a-4cc5-8769-3661fd239e98 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f3ea1580c56f4814a92bbb2acf542905', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4562f3951fbe465280073e60a11e44e2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:17:54.048763 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f95e12c-517a-4cc5-8769-3661fd239e98 service neutron] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:17:54.049118 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f95e12c-517a-4cc5-8769-3661fd239e98 service neutron] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:17:54.049523 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f95e12c-517a-4cc5-8769-3661fd239e98 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:17:54.051329 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-8f95e12c-517a-4cc5-8769-3661fd239e98 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:17:54.065630 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f95e12c-517a-4cc5-8769-3661fd239e98 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:17:54.067067 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f95e12c-517a-4cc5-8769-3661fd239e98 service neutron] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:17:54.078870 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1467/2932] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Aug 30 02:17:54 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 50 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 02:17:55.262666 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0dce2aec-e339-4a7a-84d8-3aff288851a2 service nova] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:17:55.272901 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0dce2aec-e339-4a7a-84d8-3aff288851a2 service nova] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:17:55.276609 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b773939b-1425-4128-a066-aa2a14b9a1a2 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f3ea1580c56f4814a92bbb2acf542905', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4562f3951fbe465280073e60a11e44e2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:17:55.277206 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b773939b-1425-4128-a066-aa2a14b9a1a2 service neutron] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:17:55.277514 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b773939b-1425-4128-a066-aa2a14b9a1a2 service neutron] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:17:55.277790 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b773939b-1425-4128-a066-aa2a14b9a1a2 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:17:55.280414 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-b773939b-1425-4128-a066-aa2a14b9a1a2 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:17:55.293711 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b773939b-1425-4128-a066-aa2a14b9a1a2 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:17:55.295307 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b773939b-1425-4128-a066-aa2a14b9a1a2 service neutron] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:17:55.306245 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1466/2933] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Aug 30 02:17:55 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 44 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 02:18:04.073205 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=6 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:18:05.301374 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=6 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:18:07.007736 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f95e12c-517a-4cc5-8769-3661fd239e98 service neutron] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:18:07.016759 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f95e12c-517a-4cc5-8769-3661fd239e98 service neutron] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:18:07.020473 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a78cff2-722b-48a6-8916-9c99291c52e9 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f3ea1580c56f4814a92bbb2acf542905', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4562f3951fbe465280073e60a11e44e2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:18:07.021066 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a78cff2-722b-48a6-8916-9c99291c52e9 service neutron] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:18:07.021344 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a78cff2-722b-48a6-8916-9c99291c52e9 service neutron] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:18:07.021647 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a78cff2-722b-48a6-8916-9c99291c52e9 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:18:07.023358 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-7a78cff2-722b-48a6-8916-9c99291c52e9 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:18:07.085110 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a78cff2-722b-48a6-8916-9c99291c52e9 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:18:07.086565 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a78cff2-722b-48a6-8916-9c99291c52e9 service neutron] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:18:07.095879 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1468/2934] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Aug 30 02:18:07 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 89 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 02:18:10.679119 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b773939b-1425-4128-a066-aa2a14b9a1a2 service neutron] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:18:10.680759 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-b773939b-1425-4128-a066-aa2a14b9a1a2 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:18:10.695524 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b773939b-1425-4128-a066-aa2a14b9a1a2 service neutron] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:18:10.700864 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-061a26cc-1461-4e84-8c5c-27351cf6be37 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '379536ff06fd4f719899e5dde732e63d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4562f3951fbe465280073e60a11e44e2', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:18:10.701487 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-061a26cc-1461-4e84-8c5c-27351cf6be37 service placement] REQUEST_METHOD: `GET` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:18:10.702180 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-061a26cc-1461-4e84-8c5c-27351cf6be37 service placement] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:18:10.702528 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-061a26cc-1461-4e84-8c5c-27351cf6be37 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:18:10.704809 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-061a26cc-1461-4e84-8c5c-27351cf6be37 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:18:10.717237 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-061a26cc-1461-4e84-8c5c-27351cf6be37 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:18:10.719916 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-061a26cc-1461-4e84-8c5c-27351cf6be37 service placement] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:18:10.730666 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1467/2935] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Aug 30 02:18:10 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 52 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 02:18:14.467752 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a78cff2-722b-48a6-8916-9c99291c52e9 service neutron] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:18:14.475046 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-7a78cff2-722b-48a6-8916-9c99291c52e9 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:18:14.490795 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a78cff2-722b-48a6-8916-9c99291c52e9 service neutron] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:18:14.495407 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cccea78d-9a2b-4613-a09c-64fb5cb13e3d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:18:14.496035 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cccea78d-9a2b-4613-a09c-64fb5cb13e3d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:18:14.496316 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cccea78d-9a2b-4613-a09c-64fb5cb13e3d admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:18:14.496783 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cccea78d-9a2b-4613-a09c-64fb5cb13e3d admin admin] PATH_INFO: `/v3/users/40e0086c1b924e7f86bd814bb45be3d9` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:18:14.497951 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cccea78d-9a2b-4613-a09c-64fb5cb13e3d admin admin] RBAC: Authorizing `identity:delete_user(user_id=40e0086c1b924e7f86bd814bb45be3d9)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:18:14.500487 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cccea78d-9a2b-4613-a09c-64fb5cb13e3d admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:18:14.652335 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-cccea78d-9a2b-4613-a09c-64fb5cb13e3d admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '40e0086c1b924e7f86bd814bb45be3d9'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:18:14.652965 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-cccea78d-9a2b-4613-a09c-64fb5cb13e3d admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '40e0086c1b924e7f86bd814bb45be3d9'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:18:14.653591 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-cccea78d-9a2b-4613-a09c-64fb5cb13e3d admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '40e0086c1b924e7f86bd814bb45be3d9'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:18:14.666429 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-cccea78d-9a2b-4613-a09c-64fb5cb13e3d admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '40e0086c1b924e7f86bd814bb45be3d9'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:18:14.673227 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-cccea78d-9a2b-4613-a09c-64fb5cb13e3d admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '40e0086c1b924e7f86bd814bb45be3d9'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:18:14.764736 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1469/2936] 10.222.0.27 () {58 vars in 1224 bytes} [Tue Aug 30 02:18:14 2022] DELETE /identity/v3/users/40e0086c1b924e7f86bd814bb45be3d9 => generated 0 bytes in 297 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 02:18:15.045437 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-061a26cc-1461-4e84-8c5c-27351cf6be37 service placement] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:18:15.058610 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-061a26cc-1461-4e84-8c5c-27351cf6be37 service placement] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:18:15.117552 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2a3cd1c-d36f-48e7-b705-c6e4c6976406 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:18:15.118212 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2a3cd1c-d36f-48e7-b705-c6e4c6976406 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:18:15.118519 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2a3cd1c-d36f-48e7-b705-c6e4c6976406 admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:18:15.118788 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2a3cd1c-d36f-48e7-b705-c6e4c6976406 admin admin] PATH_INFO: `/v3/projects/fa2e7ced1148461faedcd0e1856bae95` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:18:15.119921 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2a3cd1c-d36f-48e7-b705-c6e4c6976406 admin admin] RBAC: Authorizing `identity:delete_project(project_id=fa2e7ced1148461faedcd0e1856bae95)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:18:15.121840 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2a3cd1c-d36f-48e7-b705-c6e4c6976406 admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:18:15.227486 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-c2a3cd1c-d36f-48e7-b705-c6e4c6976406 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'fa2e7ced1148461faedcd0e1856bae95', 'request_id': 'req-c2a3cd1c-d36f-48e7-b705-c6e4c6976406'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:18:15.233109 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-c2a3cd1c-d36f-48e7-b705-c6e4c6976406 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'fa2e7ced1148461faedcd0e1856bae95', 'request_id': 'req-c2a3cd1c-d36f-48e7-b705-c6e4c6976406'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:18:15.240535 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-c2a3cd1c-d36f-48e7-b705-c6e4c6976406 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'fa2e7ced1148461faedcd0e1856bae95', 'request_id': 'req-c2a3cd1c-d36f-48e7-b705-c6e4c6976406'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:18:15.318506 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-c2a3cd1c-d36f-48e7-b705-c6e4c6976406 admin admin] The token cache is being invalidate because project fa2e7ced1148461faedcd0e1856bae95 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81578) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 30 02:18:15.318794 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-c2a3cd1c-d36f-48e7-b705-c6e4c6976406 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:18:15.319405 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-c2a3cd1c-d36f-48e7-b705-c6e4c6976406 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:18:15.320877 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1468/2937] 10.222.0.27 () {58 vars in 1233 bytes} [Tue Aug 30 02:18:15 2022] DELETE /identity/v3/projects/fa2e7ced1148461faedcd0e1856bae95 => generated 0 bytes in 276 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 30 02:18:15.325025 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cccea78d-9a2b-4613-a09c-64fb5cb13e3d admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:18:15.326479 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-cccea78d-9a2b-4613-a09c-64fb5cb13e3d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:18:15.339693 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cccea78d-9a2b-4613-a09c-64fb5cb13e3d admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:18:15.397664 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d204d136-f085-4af7-9670-bd8c39a94779 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:18:15.398296 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d204d136-f085-4af7-9670-bd8c39a94779 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:18:15.398579 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d204d136-f085-4af7-9670-bd8c39a94779 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:18:15.398856 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d204d136-f085-4af7-9670-bd8c39a94779 admin admin] PATH_INFO: `/v3/users/4c2c70735cff4748866f0dab613d24a0` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:18:15.399985 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d204d136-f085-4af7-9670-bd8c39a94779 admin admin] RBAC: Authorizing `identity:delete_user(user_id=4c2c70735cff4748866f0dab613d24a0)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:18:15.401942 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d204d136-f085-4af7-9670-bd8c39a94779 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:18:15.407840 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:18:15.584630 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-d204d136-f085-4af7-9670-bd8c39a94779 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '4c2c70735cff4748866f0dab613d24a0'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:18:15.585281 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-d204d136-f085-4af7-9670-bd8c39a94779 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '4c2c70735cff4748866f0dab613d24a0'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:18:15.585944 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-d204d136-f085-4af7-9670-bd8c39a94779 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '4c2c70735cff4748866f0dab613d24a0'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:18:15.597929 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-d204d136-f085-4af7-9670-bd8c39a94779 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '4c2c70735cff4748866f0dab613d24a0'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:18:15.604614 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-d204d136-f085-4af7-9670-bd8c39a94779 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '4c2c70735cff4748866f0dab613d24a0'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:18:15.683883 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1470/2938] 10.222.0.27 () {58 vars in 1224 bytes} [Tue Aug 30 02:18:15 2022] DELETE /identity/v3/users/4c2c70735cff4748866f0dab613d24a0 => generated 0 bytes in 359 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 02:18:15.948024 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2a3cd1c-d36f-48e7-b705-c6e4c6976406 admin admin] Authenticating user token {{(pid=81578) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:18:15.949442 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.fernet_utils [None req-c2a3cd1c-d36f-48e7-b705-c6e4c6976406 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81578) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:18:15.963580 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2a3cd1c-d36f-48e7-b705-c6e4c6976406 admin admin] Validating token access rules against request {{(pid=81578) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:18:16.033700 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-504e63cf-5963-4afb-9077-997110266d43 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac83d5ed111c4f4e8b3c3226bd3bbaba', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '70bc9f42895844308fb4b9192314f80b', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81578) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:18:16.034339 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-504e63cf-5963-4afb-9077-997110266d43 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:18:16.034621 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-504e63cf-5963-4afb-9077-997110266d43 admin admin] SCRIPT_NAME: `/identity` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:18:16.034896 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.server.flask.request_processing.req_logging [None req-504e63cf-5963-4afb-9077-997110266d43 admin admin] PATH_INFO: `/v3/projects/821c7a0aa57a472398a82afbc16a598e` {{(pid=81578) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:18:16.036012 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-504e63cf-5963-4afb-9077-997110266d43 admin admin] RBAC: Authorizing `identity:delete_project(project_id=821c7a0aa57a472398a82afbc16a598e)` {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:18:16.037949 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-504e63cf-5963-4afb-9077-997110266d43 admin admin] RBAC: Authorization granted {{(pid=81578) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:18:16.058539 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:18:16.136089 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-504e63cf-5963-4afb-9077-997110266d43 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '821c7a0aa57a472398a82afbc16a598e', 'request_id': 'req-504e63cf-5963-4afb-9077-997110266d43'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:18:16.142199 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-504e63cf-5963-4afb-9077-997110266d43 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '821c7a0aa57a472398a82afbc16a598e', 'request_id': 'req-504e63cf-5963-4afb-9077-997110266d43'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:18:16.148369 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-504e63cf-5963-4afb-9077-997110266d43 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '821c7a0aa57a472398a82afbc16a598e', 'request_id': 'req-504e63cf-5963-4afb-9077-997110266d43'} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:18:16.222739 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-504e63cf-5963-4afb-9077-997110266d43 admin admin] The token cache is being invalidate because project 821c7a0aa57a472398a82afbc16a598e was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81578) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 30 02:18:16.222980 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-504e63cf-5963-4afb-9077-997110266d43 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:18:16.223595 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG keystone.notifications [None req-504e63cf-5963-4afb-9077-997110266d43 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81578) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 30 02:18:16.225092 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: [pid: 81578|app: 0|req: 1469/2939] 10.222.0.27 () {58 vars in 1233 bytes} [Tue Aug 30 02:18:15 2022] DELETE /identity/v3/projects/821c7a0aa57a472398a82afbc16a598e => generated 0 bytes in 278 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 30 02:18:25.680288 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=23,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:18:26.218204 dv-cinder-819667-6-iscsi devstack@keystone.service[81578]: DEBUG dbcounter [-] [81578] Writing DB stats keystone:SELECT=16,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81578) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 30 02:20:24.268778 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d204d136-f085-4af7-9670-bd8c39a94779 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 30 02:20:24.270197 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-d204d136-f085-4af7-9670-bd8c39a94779 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:20:24.410024 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d204d136-f085-4af7-9670-bd8c39a94779 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 30 02:20:24.466622 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3836538c-4e1a-4978-86fa-b797ff93e2ae service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '379536ff06fd4f719899e5dde732e63d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '4562f3951fbe465280073e60a11e44e2', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 30 02:20:24.467221 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3836538c-4e1a-4978-86fa-b797ff93e2ae service placement] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 30 02:20:24.467498 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3836538c-4e1a-4978-86fa-b797ff93e2ae service placement] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 30 02:20:24.467766 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3836538c-4e1a-4978-86fa-b797ff93e2ae service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 30 02:20:24.469450 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-3836538c-4e1a-4978-86fa-b797ff93e2ae service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 30 02:20:24.480346 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3836538c-4e1a-4978-86fa-b797ff93e2ae service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 30 02:20:24.481948 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3836538c-4e1a-4978-86fa-b797ff93e2ae service placement] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 30 02:20:24.492320 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1471/2940] 10.222.0.27 () {60 vars in 1365 bytes} [Tue Aug 30 02:20:24 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 224 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 30 02:20:34.488161 dv-cinder-819667-6-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=89 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}}